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FOREWORD 


Historically,  Electronic  Warfare  (EW)  Systems  have  been  complex  and  frequently 
present  a  dichotomy  because  they  represent  high  technology  at  its  best  and  its  worst.  At 
its  best,  EW  can  be  a  major  technical  and  tactical  edge  (force-multiplier)  over  potential 
adversaries  (as  demonstrated  during  recent  military  operations);  while  at  its  worst,  EW 
can  incur  vicious  spiraling  costs  and  can  push  risk  and  technological  state-of-the-art  to 
keep  pace  with,  or  exceed,  technical  advances  in  potential  threat  systems  and  their 
counter-countermeasures  (CCM).  Needs  for  new  EW  capabilities  arise  usually  in 
response  to  known  or  expected  actions  by  a  potentially  hostile  entity  about  which  little 
may  be  known.  The  need  for  rapid,  decisive  response  can  cause  the  acquisition  process 
for  new  EW  capabilities  to  be  less  conventional  and  more  dynamic  than  other  types  of 
major  systems.  As  development  proceeds,  additional  intelligence  may  become  available 
that  may  cause  abrupt  changes,  or  as  a  minimum  a  re-analysis  of  the  requirements,  i.e., 
has  the  threat  changed,  expanded,  or  increased  in  complexity  enough  to  make  our  new 
capability  ineffective  when  fielded?  Because  EW  is  so  dependent  upon  the  use  of 
current,  and  in  many  cases,  highly  sensitive  intelligence  sources  and  methods,  the 
openness  of  information  relating  to  new  EW  capabilities  must  carry  the  same  level  of 
protection  to  prevent  revealing  details  of  the  capability. 

EW  systems,  to  be  effective,  have  to  be  flexible  in  responding  to  a  multiple- 
threat,  multispectral  environment  which,  in  turn,  leads  to  complex  EW  systems.  Trends 
indicate  that  current  and  future  EW  systems  will  be  required  to  be  integrated  with  or 
carried  by  a  variety  of  platforms.  New  capabilities  must  respond  in  as  close  to  real-time 
(nanoseconds)  as  technically  and  operationally  feasible.  This  infers  that  the  systems  must 
keep  pace  with  the  state-of-the-art  in  not  only  EW  technologies,  but  also  in  the 
technologies  they  must  counter,  e.g.,  low-probability-of-intercept,  low  observables,  high- 
power  microwaves. 

EW  systems  should  bring  added  value  to  the  warfighter.  The  effectiveness 
of  EW  is  difficult  to  measixre  but  can  be  tied  directly  to  the  survivability  of  the 
assets  it  is  intended  to  protect.  It  should  be  as  non-intrusive  as  possible  on  the 
warfighting  itself.  Because  the  art  of  EW  has  some  unconventional  characteristics 
and  demands,  its  conformance  to  accepted  acquisition  practices  has  often  been 
outside  the  perceived  norm.  However,  from  a  T&E  process  point  of  view,  EW 
systems  development  is  not  so  different  or  special  that,  with  the  proper 
precautions  in  place  to  prevent  premature  revelation  of  capability,  it  cannot  be 
adapted  to  a  standardized  DoD  T&E  process.  The  T&E  process  for  EW  systems 
described  in  this  document  represents  a  compendium  of  best  T&E  practices  and 
procedures  and  is  designed  to  allow  that  conformance  to  be  achieved. 


TABLE  OF  CONTENTS 


FOREWORD 

TABLE  OF  CONTENTS . iii 

LIST  OF  FIGURES . vi 

1.  INTRODUCTION . 1 

1.1.  Objective . 1 

1.2  Background . 1 

1.3  Terminology . 1 

2.  THE  DOD  TEST  AND  EVALUATION  PROCESS  FOR  EW  SYSTEMS . 4 

2.1  Overview . 4 

2.2  General . 5 

2.3  T&E  Coordination  Function . 5 

2.4  TifeE  Implementation  Fimction . 123- 

3.  EW  SYSTEMS  T&E  DURING  PHASE  0  -  CONCEPT  EXPLORATION  AND 

DEFINITION-(CE&D) . 1924 

3.1  STEP  ONE.  Identification  of  Information  Needed  by  Decision  Maker . 202 

3.2  STEP  TWO.  Pre-test  Analysis . 212 

3.3  STEP  THREE.  Test  Activity  and  Data  Management . 212 

3.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation . 232 

3.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 

Program  Information . 232 

4.  EW  SYSTEMS  T&E  DURING  PHASE  I  DEMONSTP^TION  AND 

VAT  JDATTON  IDEMWAL^  PROGRAM  DEFINITION  AND  RISK 
REDUCTION . 242 

4. 1  STEP  ONE.  Identification  of  Information  Needed  by  Decision  Maker . 262 

4.2  STEP  TWO.  Pre-test  Analysis . 27^ 


iii2222 


4.3  STEP  THREE.  Test  Activity  and  Data  Management . 27^ 

4.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation . 29^ 

4.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 

Program  Information . 29^ 

5.  EW  SYSTEMS  T&E  DURING  PHASE  II  -  ENGINEERING  AND 

MANUFACTURING  DEVELOPMENT  (EMD) . 114^ 

5.1  STEP  ONE.  Identification  of  Information  Needed  by  Decision  Maker . 324 

5.2  STEP  TWO.  Pre-test  Analysis . 33^ 

5.3  STEP  THREE.  Test  Activity  and  Data  Management . 335- 

5.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation . 356 

5.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 

Program  Information . 36 

6.  SYSTEMS  T&E  DURING  PHASE  III  -  PRODUCTION,  FIELDING  AND 

DEPLOYMENT  (PF&D) . 3840 

6. 1  STEP  ONE.  Identification  of  Information  Needed  by  Decision  Maker 3944 

6.2  STEP  TWO.  Pre-test  Analysis . 3945 

6.3  STEP  THREE.  Test  Activity  and  Data  Management . 405 

6.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation . 415 

6.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 

Program  Information . 415 

7.  EW  SYSTEMS  T&E  DUPJNG  PIL\SE  IV  OPERATIONS  AND  SUPPORT- 

(O&S) . ^5 

- 7.1  Milestone  IV  Major  Modification  Approval . . ^6 

- 7.2  Phase  IV  Operationo  and  Support . ‘16 

- 7.3  EW  T&E  Process  Applied  to  Modifications/Upgrades . i. . ^7 

75.  SUMMARY . 425 

APPENDIX  A  -  ABBREVIATIONS  AND  ACRONYMS . A-1 

APPENDIX  B  -  DEFINITIONS . B-1 

REPORT  DOCUMENTATION  PAGE 


iv5556 


LIST  OF  FIGURES 


Figure . 

1 .  The  Five  Step  T&E  Process  for  EW  Systems . 

2.  Example  RDA  Activities . 

3.  Step  One:  Identification  of  Information  Needed  by  Decision  Maker . 

4.  Sample  Missile  Warning  System  Requirements  Correlation 

Matrix  (RCM) . 

5.  Sample  Missile  Warning  System  Maturity  Matrix  (SMM) . 

6.  Step  Five:  Decision  Maker  Weighs  the  T&E  Information  against  Other 

Program  Information . 

7.  Step  Two:  Pre-Test  Analysis . 

8.  Step  Three:  Test  Activity  and  Data  Management . 

9.  Step  Four:  Post  Test  Synthesis  and  Evaluation . 

1 0.  Phase  0  T&E  Questions . 

1 1 .  Phase  I  T &E  Questions . 

12.  Phase  II  T&E  Questions . 

1 3 .  Phase  III  T&E  Questions . 

— 14^ — Phase  IV  T&E  Questions . : . 


Page 

. 4 

. 66 

..940 

. 11 

. 12 

..123- 

..134 

..134 

..180 

..202 

..260 

..324 

3944 


v332§ 


A  DESCRIPTION  OF  THE  DOD  TEST  AND  EVALUATION  PROCESS 

FOR 

ELECTRONIC  WARFARE  SYSTEMS 


1.  INTRODUCTION 

1.1  Objective.  The  principal  objective  of  this  document  is  to  describe  a  rigorous, 
standardized,  disciplined  test  and  evaluation  (T&E)  process  for  Electronic  Warfare  (EW) 
systems.  The  Process  is  comprised  of  functions  and  steps  to  assist  the  decision  making 
process  leading  to  progressive  acquisition  of  mission-required  EW  systems. 

1.2  Background. 

1.2.1  In  mid-1993,  the  Under  Secretary  of  Defense  (Acquisition  & 
Technology),  Director,  Test  and  Evaluation  convened  a  task  force  to  develop  a  process 
for  EW  systems  T&E.  This  task  force,  with  the  concurrence  of  their  Service  Executives, 
produced  a  DoD  T&E  Process  for  EW  systems  described  in  this  document.  DoD 
Instruction  5000.2,  Defense  Acquisition  Policies  and  Procedures,  was  used  as  the 
baseline  for  this  process.  That  is,  DoDI  5000.2  provideds  the  framework  around  this  DoD 
T&E  Process  for  EW  systems.  This  revision  reflects  the  changes  in  the  acquisition 
process  incorporated  in  the  new  DoD  5000.2-R.  Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated  Information  System 
fMAIS)  Acquisition  Programs.  This  document  supports  DoD  5000.2-R  DoDI  5000:2  and 
is  not  in  conflict  with  the  5000  series  of  documents. 

1.2.2  Specific  program  reporting  requirements  are  announced  separately 
by  DT&E  and  OT&E.  The  T&E  process  described  herein  may  be  applied  to  EW  systems 
that  conform  to  the  definition  in  Paragraph  1.3.1  of  this  document.  This  definition  is  from 
CJCS  Memorandum  of  Policy  (MOP)  No.  30  that  promulgates  Command  and  Control 
Warfare  (C2W)  as  a  military  strategy.  In  addition  to  establishing  C2W  as  a  new  strategy, 
MOP  30  effectively  eliminates  Command,  Control,  Communications  Countermeasures 
(C3CM)  as  a  military  philosophy  and  a  working  definition  by  replacing  it  with  C2W  (see 
Paragraph  1.3.2).  Two  other  changes  promulgated  by  MOP  30  in  the  shift  from  C3CM  to 
C2W  are:  1)  to  change  the  term  "jamming"  to  the  much  broader  military  action  of  EW 
and  2)  to  add  psychological  operations  (PSYOP)  as  a  principal  military  action.  The  latter 
(along  with  operational  security  (OPSEC))  is  not  addressed  in  this  document. 

1.3  Terminology. 

1.3.1  Electronic  Warfare  TEW).  Electronic  Warfare  is  a  military  action 
involving:  1)  the  use  of  electromagnetic  or  directed  energy  to  attack  an  enemy's  combat 
capability,  2)  protection  of  friendly  combat  capability  against  undesirable  effects  of 
friendly  or  enemy  employment  of  EW,  or  3)  surveillance  of  the  electromagnetic  spectrum 
(also  known  as  EW  Support)  for  immediate  threat  recognition  in  support  of  electronic 


warfare  operations  and  other  tactical  actions  such  as  threat  avoidance,  targeting,  and 
homing. 


1.3. 1.1  Electronic  Attack  (EAV  Involves  the  use  of 
electromagnetic  or  directed  energy  to  attack  personnel,  facilities,  or  equipment  vvith  the 
intent  of  degrading,  neutralizing,  or  destroying  enemy  combat  capability.  Representative 
examples  include,  but  are  not  limited  to:  jamming,  e.g.,  noise,  repeater;  deception 
countermeasures  (DECM),  e.g.,  false  targets,  range  denial,  velocity  (Doppler)  denial; 
physical  kill,  e.g.,  high-power  microwaves. 

1 .3 . 1 .2  Electronic  Protection  (EPI.  Involves  the  actions  taken  to 
protect  personnel,  facilities,  and  equipment  from  any  effects  of  friendly  or  enemy 
employment  of  EW  that  degrade,  neutralize  or  destroy  friendly  combat  capability. 
Representative  examples  include,  but  are  not  limited  to:  automatic  warning  sensors,  e.g. 
radar  homing  and  warning  (RHAW),  missile  attack  warning,  laser  warning; 
decoys/RPVs;  expendables,  e.g.,  chaff,  flares  (IRCM);  electronic  coimter- 
countermeasures  (ECCM),  e.g.,  antijam  techniques,  radar-cross-section  (RCS)  reduction, 
wartime  reserve  modes  (WARM),  and  anti-air,  anti-radiation  missile  (AARM). 

1.3. 1.3  Electronic  Support  (ES).  Involves  actions  tasked  by,  or 
under  direct  control  of,  an  operational  commander  to  search  for,  intercept,  identify,  and 
locate  sources  of  intentional  and  unintentional  radiated  electromagnetic  energy  for  the 
purpose  of  immediate  threat  recognition.  Representative  examples  include,  but  are  not 
limited  to:  detection,  direction  finding  (DF),  precision  emitter  location,  signal  parametric 
analysis,  threat  system  classification. 

1 .3.2  Command  and  Control  Warfare  (C2W).  The  integrated  use  of 
OPSEC,  military  deception,  PSYOP,  EW,  and  physical  destruction,  mutually  supported 
by  intelligence,  to  deny  information  to,  influence,  degrade,  or  destroy  adversary  C2 
capabilities,  while  protecting  friendly  C2  capabilities  against  such  actions. 

1.3.3  DoD  Annual  EW  Plan.  The  annual  DoD  EW  Plan  compiles  EW, 
tactical  cryptologic,  and  tactical  SIGESIT  technology  (multi-year)  RDT&E  and  production 
plans  in  the  President's  annual  budget  submitted  to  the  Congress.  The  plan  is  prepared  by 
the  Joint  Command  and  Control  Warfare  Center  (JC2WC),  the  Services,  the  Defense 
Intelligence  Agency  (DIA),  National  Security  Agency  (NSA),  and  the  Joint  Staff  (JS) 
under  the  direction  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
[USD(A&T)]. 


1.3.4  T&E  Oversight.  Programs  under  OSD  T&E  oversight  are  subject  to 
the  provisions  of  the  DoD  5000  series  which  require  that  each  program  have  a  Test  and 
Evaluation  Master  Plan  (TEMP)  and  that  OSD  approve  the  TEMP.  The  DoD  5000.2-R 
DoDI  5000:2  requires  that  the  DOT&E  and  the  DT&E  jointly  publish  an  annual  listing  of 
major  and  other  designated  defense  acquisition  programs  for  OSD  T&E  oversight.  EW 
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systems  xmder  OSD  T&E  oversight  include  all  ACAT I  programs  as  well  as  selected 
ACAT II  and  III  programs. 

1.3.5  Annual  Report  to  Congress.  A  consolidated  DoD  report  on  the 
implementation  of  this  process  bv  DoD  Components  for  those  EW  systems  under  OSD 
T&E  oversight  will  be  included  in  the  Annual  DoD  EW  Plan  each  year.  Based  on 
Service  program  innuts,  the  Director.  Test.  Systems  Engineering  and  Evaluation 
OUSPrA&TI.  with  the  Director.  Operational  Test  and  Evaluation,  will  prepare  the 
ioiatPoD  annual  -renort-fei^ftelusion  in  the  annual  EW^plaft.T 

_ Appendix  A  contains  a  list  of  abbreviations  and  acronyms  used  in  this  document. 

Appendix  B  contains  the  definition  of  other  significant  terms  used  in  this 
document. 


2.  THE  DOD  TEST  AND  EVALUATION  PROCESS  FOR  EW  SYSTEMS 


2.1  Overview.  A  DoD  Test  and  Evaluation  (T&E)  Process  for  Electronic  Warfare 
Systems,  shown  in  Figure  1,  is  a  five  step  iterative  process  that  provides  answers  to 
critical  EW  T&E  questions  for  decision  makers  to  support  decisions  for  each  phase  of 
the  acquisition  process. 


TEST  &  EVALUATION  PROCESS 

T&E  COORDINATION  FUNCTION 


T&E  IMPLEMENTATION  FUNCTION 
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MODELS  /  SIMULATION  SYSTEMS  USED  THROUGHOUT 


Figure  1.  The  Five  Step  T&E  Process  for  EW  Systems 


The  Process  is  contained  within  two  major  functions,  the  T&E  Coordination  Function 
and  the  T&E  Implementation  Function.  The  Process  can  involve  extensive  use  of 
modeling  and  simulation  systems.  It  provides  a  continuous  approach  to  the  problem  of 
managing  EW  system  acquisition  during  T&E.  Detailed  feedback  is  essential  throughout 
the  process  for  successful  EW  system  acquisition.  The  Process  establishes  a 
comprehensive  approach  to  T&E  that  may  be  applied  one  or  more  times  within  each  of 
the  five-four  phases  of  the  DoD  acquisition  process. 


The  process  is  structured  so  that  it  can  be  used  at  any  time  by  any  program.  Its 
application  and  execution  are  based  upon  EW  T&E  information  needed  by  decision 
makers. 


2.2  General.  After  JROC  validation  of  the  mission  need  (for  ACAT  1  and  lA 
prnpramsV  the  USD(A&T)  convenes  a  ^Fhe-Milestone  0  (MS  0)  Defense  Acquisition 
Board  (DAB')  to  review  the  deei&ion  marks  the  beginning  of  the  acquisition  process  with 
the  approval  of  the  Mmission  j4needs -Sstatement  (MNS).  identify  possible  materiel 
alternatives,  and  authorize  concept  studies  if  deemed  necessary.  _The  EW  MNS  defines 
projected  needs  for  an  EW  capability  in  broad  operational  terms  of  mission  objectives 
and  general  capabilities  providing  a  clear  military  worth.  A  favorable  Milestone  0 
decision  aotivate.s  the  DoD  EW  T&E  process  but  does  not  mean  that  a  new  acquisition 
program  has  been  initiated.  The  MS  0  decision  also  identifies  the- alternative  EW 
concepts  or  designs  to  be  studied;  approves  the  start  of  Phase  0 — Concept  Exploration 
and  Definition,  establishes  Phase  0  exit  criteria  to  be  evaluated  at  Milestone  E  tmd 
activates  the  DoD  T&E  Process.  Milestone  decisions  are  documented  in  the  Acquisition 
Decision  Memorandum  (ADM)  for  Acquisition  Category  (ACAT)  I  EW  programs. 

2.2.1  During  each  phase  and  at  each  milestone,  decision  makers  need 
information  from  several  sources  to  determine  the  best  course  of  action.  The 
information  needed  from  T«&E  can  be  summarized  as  a  series  of  questions  to  be 
answered  during  the  process  in  each  phase.  The  questions  are  the  basis  for  the  T&E 
process  in  that  phase;  and,  in  this  document,  they  are  stated  in  the  context  of  the 
Process  as  they  relate  to  each  phase.  The  outcome  of  the  EW  T&E  process  is  technical 
and  operational  assessments  delivered  to  the  decision  maker  as  answers  to  those 
questions. 

2.3  T&E  Coordination  Function.  The  T&E  process  for  EW  systems  begins  with 
the  T&E  Coordination  Function,  which  has  three  parts:  the  Research,  Development  and 
Acquisition  (RDA)  Function  and  Steps  One  and  Five  of  the  Process. 

2.3.1  Research.  Development  and  Acquisition  rRDA)  Function  activities  are  the 
basis  for  T&E  activities  in  that  the  MNS  and  Operational  Requirements  Document 
(ORD)  contain  the  requirements  against  which  the  EW  system  must  be  evaluated  (see 
examples  in  Figure  3).  Although  requirements  generation  and  maintenance  are  not  part  of 
the  T&E  process,  the  T&E  community  should  be  involved  in  the  generation  of  these 
requirements  to  ensure  that  identified  requirements  are  testable,  measurable  and  can  be 
evaluated.  Other  RDA  activities  that  affect  T&E  range  from  specification  identification 
through  design  and  development  and  production  to  training  and  support  needs.  The 
degree  to  which  each  of  these  has  been  demonstrated  must  be  considered  by  decision 
makers.  Answers  with  regard  to  the  achievement  of  EW  system  performance  objectives 
come  from  T&E.  Answers  to  other  questions,  such  as  urgency,  military  effectiveness, 
costs  and  schedules,  come  from  other  sources  such  as  Intelligence,  Program  Analysis  and 
Evaluation  (PA&E),  Legal,  or  Comptroller. 

2. 3. 1.1  Successful  EW  T&E  is  driven  by  the  operational  environment  and 
the  military  mission  that  must  be  accomplished.  The  dominant  measurement  of  the 
operational  utility  is  how  well  the  EW  system  is  able  to  perform  its  operational  mission 
over  time.  At  MS  0,  the  user  presents  the  MNS  to  the  Defense  Acquisition  Board 
(DAB).  If  the  MNS  receives  favorable  consideration,  the  DAB  authorizes  entry  into_ 
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The  focus  of  Phase  0,  Concept  Exploration,  efforts  and  Definition,  is  to  define  and 
evaluate  the  feasibility  of  alternative  concents  and  to  nrovde  a  basis  for  assessing  the 
relative  merits  (i.e..  advantages  and  disadvantages,  degree  of  risk)  of  these  concepts  at 
the  next  milestone  decision  point.  An  analysis  of  alternatives  is  used  as  appropriate  to 
compare  alternative  concepts.  The  most  promising  system  concepts  shall  be  defined  in 
terms  of  initial,  hroad.  objectives  for  cost,  schedule,  performance,  software 
requirements,  opportunities  for  tradeoffs,  overall  acqusisition  strate2v.  and  test  and 
evaluation  strategy.  Test  and  evaluation  planning  shall  begin  in  Phase  0.  Both 
developmental  and  operational  testers  shall  be  involved  early  to  ensure  that  the  test 
program  for  the  most  promising  alternative  can  support  the  acquisition  strategy  and  to 
ensure  the  harmonization  of  objectives,  thresholds,  and  measures  of  effectiveness 
tMOEsI  in  the  ORD  and  TEMP.  Test  and  evaluation  planning  shall  address  MOEs  and 
measures  of  performance  (MOPs^  with  appropriate  quantitative  criteria,  test  event  or 
scenario  description,  resource  requirements  (e.g..  special  instrumentation,  test  articles, 
validated  threat  targets,  validated  threat  simulators  and  validated  threat  simulations, 
actiial  threat  svstems  or  surrogates,  and  personnel^,  and  identify  test  limitations-feem 
which  are  formulated  the  operational  suitability  and  effectiveness  parameters  and 
critical  technical  parameters  (CTPs)  against  which  the  EW  system  is  tested  and 
evaluated.  The  objectives  of  Phase  0  are  to  explore  various  materiel  alternatives  to 
satisfying-the  documented  mission  need,  define  the  most  promising  system  concept(s); 
develop  risk  analyses,  and  develop  a  proposed  acquisition  strategy  and  initial  program 
objective  for  cost,  schedule  and  performance  for  the  most-promising  system  concept(B). 
A  Cost  and  Operational  Effectiveness  Analysis  (COEA)  and  ORD  are  prepared. 

As  the  acquisition  process  proceeds,  the  concepts  to  be  studied  in  Phase  0 
are  defined  and  the  engineering  and  production  prototypes  to  be  evaluated  in  Phases  I 
and  II  are  built.  In  Phases  III  and  IV  the  EW  system,  and  its  modifications  if 
appropriate,  must  be  evaluated. 


6?^ 


2.3 . 1 .2  The  RDA  Function  activities 
list  in  Figure  2  includes  the  following: 

2.3. 1.2.1  Requirements 

Definition.  System  requirements  evolve  from  the 
I  MNS  through  the  €QEA-analvsis  of  alternatives 
and  the  ORD  to  the  Acquisition  Program  Baseline 
(APB)  and  the  Test  and  Evaluation  Master  Plan 
(TEMP).  Requirements  must  be  clear,  complete, 
consistent,  feasible,  and  evaluateable.  They  are 
derived  from  military  needs  and  stated  as  technical 
and/or  operational  suitability  and  effectiveness 
parameters,  e.g.,  antenna  characteristics, 
component  sensitivity,  operating  spectrum,  agility, 
power  levels.  The  requirements  are  articulated  as 
objectives  and  thresholds  (see  DoD  5000.2-R.  Part 
2.  Page  31.  e.g.,  minimum  required  jammer 
effective  radiated  power  (ERP), 
receiver/transmitter  bandwidths,  maximum 
dynamic  range.  They  must  be  neither  wholly 
qualitative,  which  allows  uncontrolled  personal 
opinion  to  enter  the  process,  nor  wholly  quantitative,  forcing  "failure"  of  a  system  that 
is  "good  enough"  but  not  perfect.  Military  needs  (in  the  MNS)  and  operational 
requirements  (in  the  ORD)  are  the  criteria  used  to  determine  the  military  worth  of  a 
system  relative  to  the  military  needs. 

Operational  requirements  are  a  basis  for  CTPs,  used  by  the 
developmental  T&E  (DT&E)  program  as  the  basis  for  its  tests.  Operational 
requirements  are  also  a  basis  for  Measures  of  Effectiveness  (MOEs),  (e.g.,  missile  miss 
distances,  kill  probabilities,  detection  and  tracking  ranges),  used  by  the  operational 
T&E  (OT&E)  agency  as  the  basis  for  its  determination  of  operational  effectiveness  and 
suitability.  Measures  of  Performance  (MOPs),  derived  from  MOEs  may  also  be  used 
to  support  the  determination  of  operational  effectiveness  and  suitability.  MOEs  and 
MOPs  form  the  basis  for  the  operational  evaluation  which,  in  turn,  shapes  the 
operational  tests. 

2. 3. 1.2.2  Digital  Modeling.  Digital  models,  implemented  on  computers, 
are  growing  in  importance,  use,  and  credibility.  A  digital  system  model  (DSM)  is  a 
computer  model,  or  software  equivalent,  of  a  system  under  development.  Digital 
models  used  in  RDA  activities  may  be  similar  or  identical  to  those  used  in  the  T&E 
Implementation  Function,  and  in  fact  may  be  first  created  during  Concept  Exploration 
I  and  Defmitionand  updated  and  used  as  the  EW  system  proceeds  through  development 
and  T&E.  Models  are  used  during  the  RDA  activity  to  evaluate  concept  feasibility,  to 
attempt  to  define  the  technical  limits  of  system  performance,  to  allocate  requirements 
and  functions,  to  plan  tests,  to  interpolate  test  results,  and  to  provide  a  rigorous 
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evaluation  methodology.  These  models  may  he  simple  thought  processes  or  "back-of- 
the-envelope"  estimates,  or  they  may  be  sophisticated  simulations  of  system 
performance.  But  the  causal  relationship  between  military  effectiveness  and  system 
performance  must  always  be  developed,  validated  and  documented. 

Those  digital  models  selected  for  use  in  the  EW  T&E  process  must 
be  appropriately  validated  and  certified.  Models  and  simulations  may  range  in  scope 
from  macroscopic,  i.e.,  operational  scenarios,  threat  engagements  (many-on-many), 
and  electromagnetic  interactions  among  systems,  to  a  high  degree  of  fidelity 
(microscopic),  i.e.,  detailed  technical  representation  of  electromagnetic  propagation  and 
signal  processing  functions  of  the  system  under  test  (SUT).  Any  or  all  of  these  types  of 
models  and  simulations  may  be  used  to  support  the  process  of  translating  the  broad 
operational  capabilities  as  described  in  the  MNS  into  system-specific  performance 
requirements  that  may  be  demonstrated  through  T&E. 

2.3. 1.2.3  Design.  System  Design  is  the  first  step  in  creating  the 
system.  EW  system  design  is  the  process  of  converting  specifications  for  an  EW 
capability  into  a  visual  or  mathematical  representation  of  the  system.  As  the  system 
characteristics  take  form,  it  is  very  important  that  progress  on  the  system  design  is 
communicated  effectively  to  the  user  to  ensure  that  the  new  EW  capability  continues  to 
satisfy  the  operational  requirements.  System  design  is  perhaps  the  most  critical  step  in 
the  RDA  function  because,  once  built,  the  EW  system  is  much  more  difficult  to  modify 
to  correct  deficiencies  built  into  the  original  design.  Once  approved,  the  design  is 
converted  into  a  prototype  (working  model).  Testing  is  conducted  to  assess  whether  the 
EW  system  design  meets  the  requirements. 

2.3. 1.2.4  Prntntvning.  Joint  Publication  1-02  states  that  a  prototype  is 
a  model  suitable  for  evaluation  of  design,  performance,  and  production  potential.  For 
the  purpose  of  this  process,  prototypes  are  classified  as  engineering  and  production. 

2.3. 1.2.4. 1  Engineering  Prototype.  An  engineering  prototype 
(EP)  is  a  development  model  of  a  unit  that  is  close  to  production.  This  term  may  apply 
to  circuitry,  a  device  (black  box),  or  a  system,  and  may  be  in  a  breadboard  (technical) 
configuration.  EPs  are  normally  used  in  Phase  I. 

2.3.1.2.4.2  Production  Prototypes  (PP).  A  production  prototype 
is  a  final  model  of  a  design  before  the  pilot  unit  is  approved  for  production.  It  should 
be  highly  representative  of  final  equipment,  except  that  the  exact  manufacturing 
assembly  process  and  production  design  changes  may  not  yet  be  used  or  incorporated. 

It  is  suitable  for  complete  evaluation  of  its  electrical  and/or  mechanical  form  and  may 
be  in  a  brassboard  (technical  and  operational  configuration).  PPs  are  normally  used  in 
Phase  II. 


2.3. 1.2.5  Specifications  are  the  values  that  convert  requirements  into 
design  terms.  They  must  clearly  and  accurately  relate  back  to  the  technical 


requirements  of  the  system.  An  audit  trail  from  the  requirements  documents  to  the 
specifications  ensures  that  the  system,  when  built  to  the  specification,  performs  as 
intended. 


2.3. 1.2.6  Reliability  Growth  is  a  process  in  which  the  reliability  of  the 
operational  system  improves  through  identification  and  correction  of  systemic  reliability 
failures.  During  Engineering  &  Manufacturing  Development  (EMD),  laboratory  tests 
on  prototypes  are  performed  to  uncover  component  reliability  failures,  make  design 
improvements,  and  project  a  reliability  point  estimate.  Later,  the  system  is  deployed 
and  field  data  are  collected.  The  validity  of  this  process  lies  in  the  activities  associated 
with  operational  failures.  If  the  user  is  aggressive  in  reporting,  cataloging  and 
investigating  reliability  failures,  and  commits  the  time  and  resources  to  their  correction, 
reliability  will,  in  all  likelihood,  improve.  If  these  steps  are  not  taken,  reliability 
cannot  improve  and  will  probably  decrease  as  the  system  is  subjected  to  more 
maintenance  than  was  originally  intended. 

2.3. 1.2.7  Production  is  the  manufacturing  of  the  system  after  it  has 
been  funded,  found  to  be  producible,  and  deemed  operationally  suitable  and  effective. 
The  capability  and  integrity  of  the  manufacturing  process  to  produce  systems  meeting 
the  system  design  requirements  is  evaluated  and  measured  through  inspections  and 
testing.  The  attributes  of  a  producible  EW  system  design  are  that  it  can  be 
manufactured  economically  and  with  consistent  quality.  The  system  design  should 
address  the  manufacturing  facility's  variability  in  material,  process,  and  personnel 
sources  and  require  that  the  manufacturing  processes  be  controlled  to  the  level 
addressed  in  the  design. 

2.3. 1.2.8  Support  Equipment  is  that  category  of  ancillary  "things" 
necessary  to  sustain  the  EW  SUT  or,  during  T&E,  to  support,  monitor  and  record  the 
test.  Support  equipment  includes  auxiliary  power  carts,  signal  generators, 
oscilloscopes,  video  and  still  cameras,  video  and  audio  recorders,  stripchart  recorders, 
and  numerous  other  items  without  which  the  system  could  not  operate  properly  or  be 
adequately  tested.  Support  equipment  requirements  for  T&E  must  be  identified  in  Part 
V  of  the  TEMP  early  in  the  T&E  process  to  ensure  availability  when  needed. 

2.3. 1.2.9  Training  Equipment.  Since  most  new  or  highly  modified 
systems  are  somewhat  unique,  the  personnel  who  use  and  maintain  the  system  must  be 
retrained  to  use  it  properly  and  safely.  Training  equipment  includes  items  such  as 
computer  workstations,  display  devices/consoles  for  troubleshooting,  and  test  bench 
mockups  of  the  system  that  either  permit  that  training  or  are  used  during  training  to 
instruct  the  operators  and  maintainers  in  the  correct  way  to  operate  and  maintain  the 
system. 


2.3.1.2.10  Operational  Concent.  The  operational  concept,  as  defined 
in  the  ORD,  is  the  planned  methodology  by  which  the  EW  system  is  to  be  used  and 
supported  during  peace  and  conflict.  The  ORD  provides  performance  parameters  in 


terms  of  operational  suitability  and  effectiveness  criteria  thresholds  and  objectives. 
Understanding  the  operational  concept  is  critical  for  both  DT&E  and  OT&E  personnel. 

2.3.1.2.11  Logistics  Support  Concept  is  the  plan  by  which  the  EW 
system  is  maintained.  It  includes  spares,  maintenance,  transportation,  and  support 
personnel  requirements.  Developing  integrated  logistics  support  requirements 
consistent  with  readiness  objectives,  system  design  and  resources  should  be  considered 
early  in  the  acquisition  process. 


2.3.2  STEP 
ONE  is  the  identification 
of  T&E  information 
required  by  the 
Milestone  decision 
maker  (See  Figure  3). 

The  required  information 
consists  of  performance 
and  effectiveness 
evaluations  of  how  well 
the  system  meets  the 
user's  needs.  This 
information  progresses 
from  answers  concerning 
proposed  alternative  EW 
concepts  in  Phase  0  to  Figure  3.  Step  One:  Identification  of  Information  Needed 
answers  on  system  by  Decision  Maker 

technical  performance  and  operational  suitability  and  effectiveness  during  Phase  II  and 
beyond. 


2.3.2. 1  The  required  information  usually  centers  on  the  current  SUT  which 
is  in  the  form  of  concepts,  prototypes,  both  engineering  and  production,  and/or  the 
system  itself  depending  on  the  acquisition  phase.  Step  One  of  the  T&E  process  is 
initiated  by  the  preceding  Milestone  decision  of  the  Acquisition  Process. 

2. 3. 2. 2  Areas  in  which  questions  can  be  expected  from  the  Milestone 
decision  maker  in  each  phase  are:  the  relevance  of  historical  data,  accuracy  and 
connectivity  of  stated  requirements,  adequacy  of  T&E  infrastructure  and  technology 
base,  testing  alternatives,  system  performance  versus  validated  threats,  projected  impact 
of  the  system  on  battle  outcomes,  capabilities,  limitations,  feasibility,  preferred  system, 
and  T&E  exit  criteria.  Association  of  each  question  with  an  acquisition  phase  is 
discussed  later  in  the  application  of  the  T&E  Process  to  the  Acquisition  Process.  The 
principal  outcome  of  this  Step  is  the  determination  of  evaluation  objectives. 

2.3.2.2.1  To  ensure  that  a  sound,  cost-effective  approach  is  implemented, 
the  T&E  resources  need  to  be  identified  as  early  in  the  process  as  possible-preferably 
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prior  to  Step  One.  As  testing  proceeds  and  the  system  matures,  the  level  of  fidelity 
(accuracy)  of  required  data,  of  threats  and  surrogates,  and  of  the  test  environment 
increases  and  costs  escalate  rapidly.  Accordingly,  the  test  manager  should  carefully 
weigh  the  numbers  and  types  of  tests,  in  a  cost/benefits  analysis,  and  assess  required 
data  that  could  be  acquired  through  comparatively  lower  cost  modeling  and  simulation 
versus  data  that  must  be  acquired  through  the  more  costly  field  or  open-air  test  events. 

2. 3. 2.3  A  Requirements  Correlation  Matrix  (RCMf  (similar  to  the  example 
illustrated  in  Figure  4)  can  be  used  to  provide  an  audit  trail  of  the  EW  system  under 
development.  Such  a  matrix  could  include,  but  not  be  limited  to,  a  comparison  of 
operational  requirements,  operational  performance  parameters,  and  key  system 
performance  parameters.  Models  and  simulations  can  be  used  to  assist  in  establishing 
objective  and  threshold  values  which  can  be  displayed  in  an  RCM.  The  sample  RCM 
identifies  operational  requirements,  operational  performance  parameters  and  key  system 
performance  parameters.  These  characteristics  and  requirements  serve  as  the  foundation 
for  development  of  a  System  Maturity  Matrix  (SMM). 

2. 3. 2. 4  A  System  Maturity  Matrix  (SMM)  (similar  to  the  example 
illustrated  in  Figure  5)  is  an  acquisition  management  tool  that  can  be  used  to  highlight 
differences  between  the  required  objective/threshold  values  and  the  demonstrated  values 
resulting  from  scheduled  testing.  The  sample  SMM  contains  key  system  and  technical 
performance  parameter  thresholds  as  appropriate,  as  well  as  objectives,  at  specific 
points  in  time  within  the  development  process  of  the  system.  These  specific  thresholds 
are  called  "exit  criteria".  The  SMM  displays  these  parameters  necessary  to  measure 
progress  toward  meeting  the  operational  requirements.  At  the  milestone  review  points, 
decisions  are  made  as  to  the  ability  of  the  system  to  proceed  to  the  next  phase  in  the 
acquisition  process.  Exit  criteria  for  critical  parameters  are  requirements  so  important 
to  the  need  that,  if  not  satisfied,  will  result  in  cancellation  or  reassessment  of  the 
program.  The  characteristics  listed  in  the  SMM  will  likely  be  few  in  number. 
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Figure  4.  Sample  Missile  Warning  System  Requirements 
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Correlation  Matrix  (RCM) 


As  the  EW  system  progresses/matures,  sub-elements  or  new  characteristics  may  be 
added  to  these  matrices.  A  critical  characteristic  is  a  requirement  so  important  to  the 
need  that,  if  not  achieved,  will  result  in  cancellation  or  reassessment  of  a  program.  A 
threshold  is,  therefore,  the  value  a  critical  characteristic  must  meet.  It  becomes  an  exit 
criterion  when  coupled  with  a  specific  point  in  time  (a  particular  milestone  or  decision 
point)  when  it  must  be  attained.  Objectives  are  requirements,  although  not  critical,  that 
represent  user  desires  or  potential  improvement  above  thresholds.  After  T&E 
requirements  have  been  identified  and  are  considered  measurable,  pre-test  analysis  is 
performed  before  actual  testing  begins. 

2.3.3  STEP  FIVE  (See  Figure  6)  is  where  the  decision  maker  weighs  the  T&E 
information  against  other  programmatic  information  to  decide  a  proper  course  of 
action.  The  decision  will  be  based  on  criteria  of  military  worth,  as  well  as  consideration 
of  cost,  funding,  urgency,  etc.  When  associated  with  an  acquisition  milestone,  this 
decision  is  announced  in  an  Acquisition  Decision  Memorandum  (ADM)  which  outlines 
the  future  course  of  action  for  the  program  and  the  SUT.  Additionally,  each  milestone 
decision  contains  the  "exit  criteria"  for  the  next  Phase/Milestone. 
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Figure  5.  Sample  Missile  Warning  System  Maturity  Matrix  (SMM) 


2.3.4  Feedback.  It  is  possible  that  the  decision  maker  asked  the  wrong  question 
or  that  the  questions  were 
misunderstood  by  the  T&E 
community.  The  decision  maker 
should  compare  the  information 
contained  in  the  assessments  with  the 
questions  previously  asked  to  ensure 
that  the  responses  are  adequate. 

Problems  should  be  highlighted  and 
resolved.  The  decision  maker  can  then 
rephrase  succeeding  questions  to 
ensure  better  understanding  of  the 
information  needed. 


2.4  T&E  Implementation 
Function.  The  test  and  evaluation 
implementation  function  encompasses 
the  three  steps  necessary  to  develop 
the  information  needed  to  prepare  the 
assessments  used  by  decision  makers  in  Step  Five. 


T&E  Information  Against  Other  Program 
Information 
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2.4.1  STEP  TWO  (See 
Figure  7)  is  the  pre-test  analysis 
of  the  evaluation  objectives  from 
Step  One  to  determine  the  types 
and  quantities  of  data  needed,  the 
results  expected  or  anticipated 
from  the  tests,  and  the  analytical 
tools  needed  to  conduct  the  tests 
and  evaluations. 


2.4. 1.1  Pre-test 
analysis  develops  the  analytical 
tools,  allocates  test  parameters  to 
requirements,  estimates  test 
results,  determines  the  types  and 
quantities  of  data  needed,  and 
identifies  the  major  test  objectives. 

2.4. 1.2  The  use  of  validated  models  and  simulation  systems  during  pre-test 
analysis  can  aid  in  determining:  1)  how  to  design  test  scenario(s),  2)  how  to  set-up  the 
test  environment,  3)  how  to  properly  instrument  the  test,  4)  how  to  man  and  control  the 
test  resources,  5)  how  best  to  sequence  the  test  trials,  and  6)  how  to  estimate  outcomes. 
In  this  step,  models  and  simulations  are  used  to  estimate  test  results. 


2.4. 1.3  The 

end  product  of  this  step  is 
the  expected  outcome  of  the 
system  under  test.  When  a 
determination  is  made  that 
additional  data  are 
necessary  and  major  test 
objectives  are  identified,  the 
process  moves  to  Step 
Three. 


2.4.2  STEP  THREE, 
test  activity  and  data 
management.  (See  Figure  8) 
is  the  actual  test  activity 
planning,  test  conduct,  and 
data  management  practices. 

Given  the  data  requirements 
from  Step  Two,  T&E  managers  determine  what  valid  data  exist  in  historical  files  that 
can  be  applied  to  the  SUT  and  what  new  data  must  be  developed  from  test  events.  They 
plan  and  execute  the  tests  necessary  to  develop  the  data.  The  historical  and  developed 


STEP  3 


(EXAMPLE  ACTIVITIES) 

•  TEST  ACTIVITY  PLANNING 

•  TEST  CONDUCT 

•  TEST  EVENTS/DATA  TYPES; 

”  HISTORICAL 

**  COMPONENT  MEASUREMENT 
**  INTEGRATION 
**  HARDWARE-IN-THE-LOOP 
*'  INSTALLED  SYSTCM 
*•  FIELD  OR  OPEN  AIR 
“  SIMULATION/STIMULATION 

•  DATA  MANAGEMENT: 

••  COLLECTION 

••  REDUCTION 
••  ANALYSIS 
•’  AUTHENTICATION 
"  DISTRIBUTION 

•  TEST  ACTIVITY  REPORTING 
••  DEnCIENCY  REPORTING 


•  PLAN,  CONDUCT,  AND  REPORT  TESTS 

•  COLLECT,  PROCESS  &  AUTHENTICATE  DATA 

•  TEST  ACTIVITY  REPORTING 


Figure  8.  Step  Three:  Test  Activity  and  Data 
Management 


TEST  4  EVALUATIWI  PROCESS 


Figure  7.  Step  Two:  Pre-Test  Analysis 
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data  are  reviewed  for  completeness  and  accuracy,  authenticated,  and  forwarded  to  Step 
Four  for  assessment  as  measured  outcomes. 

2.4.2. 1  Test  activity  planning  includes  gathering  the  required  test  articles 
and  test  support  equipment,  scheduling  facilities  and  identifying  the  climatic,  mission 
and  threat  environments. 

2.4. 2.2  Test  conduct  uses  the  EW  T&E  capabilities  infrastructure  and  is 
the  culmination  of  test  activity  planning  and,  with  the  exception  of  historical  data,  is  the 
execution  of  the  actual  test  events.  As  shown  in  Figure  8,  test  activities  range  from 
historical  searches  of  the  performance  of  like  or  similar  components,  subsystems  and 
systems  to  actual  hardware  tests  of  component  incoming  parts  through  multi-system, 
open-air,  operationally  realistic  "free-play"  scenarios.  Data  gathered  during  the  test,  or 
from  historical  searches,  are  input  to  the  data  management  activity  for  processing. 

The  Reliance  Study  defines  EW  T&E  capabilities  as  those  resources  used 
for  the  evaluation  of  electronic  countermeasures  (ECM);  electronic  counter¬ 
countermeasures  (ECCM);  suppression  of  enemy  air  defense  (SEAD);  and  command, 
control,  communications,  and  intelligence  (C3I)  components,  subsystems,  systems,  and 
EW  functions  of  federated  or  integrated  avionic/vetronic  suites.  EW  testing  includes 
operational  test  considerations  for  missions  and  warfighting  requirements  of  the  Army, 
Navy,  Air  Force  and  Marine  Corps  and  involves  evaluation  of  their  vehicles  and  EW 
equipment  for  required  performance,  mutual  interference,  and  detectability  by  the 
enemy. 


2.4.2.2.1  Historical  Test  Data.  The  initial  step  in  any  EW  test  activity 
should  be  the  examination  of  previous  test  data  (or  descriptive  summaries)  stored  in 
historical  archives  to  ascertain  the  utility  of  previous  test  results,  i.e.,  does  the  required 
new  EW  capability  share  sufficient  commonality  with  any  predecessor  system  such  that 
some  testing  can  be  obviated.  Component,  subsystem  and  system  data  from  like  or 
similar  components,  subsystems  or  systems  must  be  examined  first  for  applicability, 
and,  if  applicable,  to  reduce  the  amount  and/or  type  of  hardware/software  testing 
necessary  for  evaluation.  Further,  modeling  and  simulation  of  historical  test  data,  in 
lieu  of  actual  testing  of  the  current  SUT,  should  be  performed  to  fill  historical 
information  voids.  This  is  a  highly  cost  effective  procedure  that  leads  to  the 
determination  of  what  data  are  lacking  and  needed  from  new  test  and  evaluation  events. 

2.4.2.2.2  Component  Measurement  Test  Events  often  involve  the  use 
of  specialized  capabilities  to  explore  and  evaluate  advanced  technologies,  e.g.. 
Microwave  Monolithic  Integrated  Circuits  (MMIC),  Very-high-speed  ICs  (VHSIC), 
Microwave  Power  Modules  (MPM),  Digital  RF  Memory  devices  (DRFM)^  devices. 
Acoustic  Charge  Transport  (ACT),  and  are  usually  the  first  test  events  performed 
during  the  development  and/or  buildup  of  the  system.  Examples  include  incoming  parts 
inspection,  thermal,  acoustic  and  vibration  cycling,  power  requirement,  and  heat 
generation  tests.  For  EW  systems,  measurement  resources  provide  those  specialized 
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capabilities  cited  above.  They  generally  fall  into  the  sub-categories  of  antenna 
measurement,  RCS  measurement,  infrared  (IR)/laser  signature  measurement,  and 
electromagnetic  interference/compatibility  (EMI/EMC)  test  capabilities. 

2.4. 2.2.3  Integration  Test  Events  test  EW  components,  subsystems  and 
systems  combined  with  other  elements.  The  other  elements  may  be  other  parts  of  the 
same  system  or  different  systems  altogether  with  which  the  SUT  must  operate.  These 
tests  are  used  to  evaluate  individual  hardware  and  software  interactions  with  each  other 
and  with  the  entire  avionic/vetronic  suite  and  are  frequently  conducted  in  integration 
laboratories  specifically  designed  to  test  the  SUT  integrated  with  other  systems  or 
functions.  Integration  laboratories  are  generally  weapon  system  specific  (except  for 
associated  environment  generators)  and  are  used  from  the  beginning  of  an  EW  system's 
development  through  avionics/vetronics  integration  and  fielding.  These  laboratories 
often  employ  a  variety  of  digital  models,  simulations,  and  stimulations  to  generate 
scenarios  and  electromagnetic  backgrounds  at  or  near  real  time. 

2.4.2.2.4  Hardware-in-the-loop  (HITL)  Events.  HITL  tests  use 
elements  of  the  SUT  in  combination  with  software  to  examine  the  performance  of  those 
elements  before  the  entire  system  is  available  or  when  a  specific  capability  cannot  be 
tested.  For  EW  systems,  these  events  are  conducted  indoors  in  a  secure  environment  to 
test  the  systems  against  manned,  closed-loop,  and  open-loop  threat  simulators.  These 
tests  provide  unique  opportunities  to  evaluate  EW  systems  hardware  at  different  stages 
of  development  (e.g.,  breadboard,  brassboard,  prototype,  or  production),  possibly 
years  before  the  host  platform  is  available,  thereby  facilitating  concurrent  development 
of  the  EW  systems  or  functions,  other  systems,  and  the  platforms/vehicles  themselves. 
HITL  testing  also  allows  production  systems  to  be  tested  under  controlled  and 
repeatable  test  conditions,  thus  providing  an  inexpensive  complement  to  flight  testing. 
Additionally,  the  controlled  environment  readily  lends  itself  to  EW  technique 
optimization  and  closed-loop  operational  effectiveness  evaluation. 

2.4. 2.2. 5  Installed  Systems  Test  Events  provide  capabilities  to  evaluate 
SUTs  and  functions  that  are  installed  on  and  integrated  with  their  host  platforms.  These 
tests  can  occur  in  indoor  facilities  such  as  EW  anechoic  chambers,  in  which  free-space 
radiation  measurements  are  made  during  simultaneous  operation  of  the  EW  system  and 
other  host  platform  systems,  or  climatic  chambers  or  as  outdoor  DT  and  OT  tests. 
Chambers  provide  secure  sites  to  evaluate  the  capabilities  and  limitations  of  the  system 
against  simulated  and  stimulated  inputs.  In  an  anechoic  chamber,  for  example,  the  SUT 
is  stimulated  by  threat  signal  generators  and  its  responses  are  evaluated  to  provide 
critical  information  regarding  integrated  system  performance.  Climatic  chambers  permit 
examination  of  SUT  capabilities  in  varied  climatic  conditions  climates  without  having  to 
transport  the  SUT  to  those  naturally  occurring  climates. 

2.4. 2.2. 6  Field  or  Open  Air  Test  Events.  The  term  "field  test"  or  open 
air  test  refers  to  any  test  conducted  outdoors.  It  includes  surface  (land  and  sea), 
undersea,  airborne  and  spacebome  testing.  Field  tests  are  conducted  where  it  is 
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feasible,  safe,  and  secure  to  test  all  or  part  of  the  SUT  in  an  environment  that  is 
normally  more  realistic  than  any  attainable  indoors.  Field  tests  may  allow  the  SUT  to 
be  operated  more  closely  to  its  operational  conditions. 

2.4. 2.2.6. 1  The  Reliance  Study  has  divided  open  air  test  resources 
into  two  subcategories:  EW  test  ranges  and  airborne  testbeds.  1)  Open  air  EW  test 
ranges  are  highly  instrumented  facilities  with  high  fidelity  threat  simulators  and  real 
systems,  and  are  primarily  used  to  test  systems  and  functions  installed  either  in  a 
testbed  or  the  intended  host  aircraft/vehicle.  Open  air  testing  provides  the  most 
operationally  realistic  environment  in  which  to  evaluate  a  SUT.  Real  world  phenomena 
encountered  during  range  testing  include  terrain  effects,  multipath  propagation, 
electromagnetic  interference  (EMI)  effects,  etc.  2)  Airborne  testbeds  are  normally 
large  airframes  designed  for  spread-bench  installation  and  testing  of  EW  and  other 
avionics  systems.  They  permit  flight  testing  of  EW  techniques,  components, 
subsystems,  systems,  or  entire  avionics  suites  in  their  early  stages  of  development 
and/or  modification,  often  prior  to  the  availability  of  prototype  or  production  hardware. 

2.4.2.2.7  .Simiilation/Stimulation  Events.  Simulation  and  stimulation 
events  are  used  extensively  in  the  DoD  test  process.  They  can  be  applied  to  computer 
or  physical  working  models  or  the  SUT.  They  may  be  real  time  or  non-real  time 
models.  Effective  use  of  credible  models  and  their  simulation/stimulation  events 
provides  cost  effective  T&E. 

2. 4. 2. 2. 7.1  According  to  the  EW  T&E  Reliance  Study,  digital 
models  and  computer  simulations  represent  EW  systems,  host  platforms,  other  friendly 
players,  the  combat  environment,  and  threat  systems.  These  models  run  interactively  in 
real  or  simulated  time  and  space  domains,  along  with  other  factors  of  a  combat 
environment.  Specific  computer  simulations  are  constructed  at  various  levels  of  detail, 
corresponding  to  the  level  of  technical  complexity  they  support  (i.e.,  engineering, 
platform,  mission,  or  campaign).  Presently,  there  are  numerous  computer  simulations 
in  use  (e.g.,  AASPEM,  SUPPRESSOR,  ALARM,  and  Enhanced  SAMS);  however, 
they  do  not  share  a  common  architecture.  For  example,  AASPEM,  the  Advanced  Air- 
to-Air  System  Performance  Evaluation  Model,  models  air-to-air  engagements,  including 
beyond-visual-range  maneuvering;  close-in-combat  air-to-air  tactics;  sensor  detection 
and  tracking;  missile  lock-on,  launch,  fly-out,  firing,  detonation,  and  kill;  gun  firing; 
laser  firing;  and  defensive  reaction  to  weapons  while  ALARM,  the  Advanced  Low 
Altitude  Radar  Model,  is  a  radar-range-equation-based  detection  model  that  includes  the 
effects  of  ground  clutter,  terrain  masking,  multipath,  diffraction,  atmospheric 
attenuation  and  jamming  against  several  types  of  radar  systems,  e.g.,  MTI,  pulse 
Doppler,  and  CW.  While  both  of  these  examples  describe  very  complex  models  that 
possess  a  high  degree  of  specificity,  their  purpose  and  use  in  the  T&E  process  would  be 
for  very  different  reasons  and  probably  at  different  phases  in  the  Acquisition  process, 
i.e.,  ALARM  would  appear  to  be  more  appropriate  for  assessing  technical  performance 
while  AASPEM  would  likely  be  used  closer  to  OT&E  to  assess  operational 
performance. 


173326 


2A22.12  A  greater  capability  to  simulate  existing  and  planned 
military  systems  and  the  threats  they  are  designed  to  counter  is  needed.  The  Joint 
Modeling  and  Simulation  System  (J-MASS)  program  is  addressing  these  problems  and 
should  be  contacted  for  use  of  their  standard  modeling  architecture  and  simulation 
support. 


2.4. 2.3  Data  Management.  Data  recorded  during  test  events  are  often  not 
in  a  form  best  suited  for  analysis.  Several  steps  are  taken  to  make  the  data  more  usable: 

2.4. 2.3.1  Data  Collection  and  Reduction.  Most  data  are  recorded 
"raw"  and  scaled  to  match  the  recording  capabilities  of  the  analog  or  digital  recording 
system  without  regard  to  the  actual  magnitude  of  the  data.  This  first  step  in  the  data 
management  process  is  assembling  data  from  all  sources  and  reducing  them  to 
engineering  values. 

2. 4. 2. 3. 2  Analysis  and  authentication  ensures  that  all  data  accurately 
reflect  the  operation  of  the  SUT.  Data  from  multiple  sources  are  compared  for 
agreement,  data  dropouts  are  filled  in  where  possible  and  questionable  data  are 
compared  with  other  sources  for  reasonableness.  The  data  are  then  distributed  for 
further  use  in  the  synthesis  and  evaluation  step. 

2.4.2.4  Deficiency  Reporting  is  the  process  of  formally  documenting 
failures  to  meet  required  performance  thresholds  or  objectives,  human  factors 
limitations,  safety  concerns,  etc.  Deficiency  reports  are  forwarded  to  the  program 
office  for  correction.  Procedures  for  deficiency  reporting  differ  among  the  Services,  but 
because  of  the  critical  importance  of  deficiency  reporting  during  field  testing,  it  should 
in  all  cases  be  a  clearly  defined,  formal  process.  Evaluation  of  the  impact  of  the 
deficiency  on  suitability  and  effectiveness  must  be  part  of  the  deficiency  reporting 
process.  Merely  quoting  specifications  as  a  justification  for  submitting  a  deficiency 
report  may  lead  to  costly,  time-consuming  changes  that  are  not  operationally  required 
or  that  preclude  making  other,  more  valuable  changes.  For  example,  the  evaluators 
should  assess  how  the  deficiency  affects  operational  mission  accomplishment.  Does  the 
deficiency  require  more  spares,  more  manpower,  or  longer  down  times  than  can  be 
afforded  in  a  military  operation?  And  finally,  can  the  user  work  around  these 
limitations  and  make  effective  use  of  the  system? 

2.4.2.5  The  end  product  of  this  step  is  measured  outcomes  in  the  form  of 
T&E  activity  reports  which  are  provided  to  the  analysts  for  Step  Four. 

2.4.3  STEP  FOUR,  post  test  synthesis  and  evaluation,  is  the  combination  of  the 
measured  outcomes  of  Step  Three  with  the  expected  outcomes  from  Step  Two, 
tempered  with  technical  and  operational  judgment.  The  output  of  Step  Four  is  the 
answers  to  the  questions  developed  at  Step  One.  See  Figure  9. 
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_ 2.4.3. 1  When  measured  outcomes  differ  from  expected  outcomes. 

fhp  tpst  rnnditinns  and  procedures  must  be  reexamined  to  determine  if  the  deviations  are 
rftal  that  is.  due  to  unexpected  performance  of  the  SUT.  or  are  caused  by  test 
limifatinns  such  as  a  lack  of  fidelity  in  computer  simulation,  non-availability  of  support 
assets,  or  less  than  fiill  system  availability.  If  the  differences  are  due  to  test  limitations, 
the  effect  of  the  limitations  must  be  evaluated,  if  possible,  and  judgment  used  to 
estimate  true  system  performance.  However,  since  this  may  involve  extrapolation  of 
test  data,  it  is  inherently  riskv.  Despite  the  additional  cost  and  time,  retesting  is  usually 
prudent. 

In  this  step,  models 
and  simulations  are 
normally  used  to  process 
test  data  and  to  evaluate 
system  performance  and 
effectiveness  using  data 
obtained  from  the  tests. 

The  assumptions  of 
tactics,  environment, 
system  performance, 
and  support  must  be 
carefully  chosen  and 
fully  described  and 
documented. 


2.4.4  Feedback. 

As  with  the  decision  maker,  feedback  is  necessary  within  the  T&E  process  to  ensure  the 
quality  of  the  output.  But  unlike  the  decision  maker,  who  needs  to  know  whether  the 
questions  have  been  answered  satisfactorily,  here  the  evaluators  must  ensure  that  the 
data  are  sufficient  to  answer  the  questions  posed  by  (or  as  understood  from)  the 
decision  maker.  Their  satisfaction  with  the  test  report  data  must  be  transmitted  as 
feedback  to  the  persons  responsible  for  the  pre-test  analysis  to  ensure  that  both  the 
current  test  and  the  process  itself  are  as  complete,  effective  and  efficient  as  possible. 

This  step  concludes  with  the  preparation  of  technical  and  operational 
assessments  which  answer  the  questions  from  the  decision  maker  at  Step  One. 

2.4.5  The  T&E  process  concludes  at  STEP  FIVE,  as  noted  above,  where  the 
decision  maker  weighs  the  T&E  information  against  other  program  information  and 
assesses  the  progress  of  the  EW  SUT. 


Figure  9.  Step  Four:  Post  Test  Synthesis  and  Evaluation 
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3.  EW  SYSTEMS  T&E  DURING  PHASE  0  -  CONCEPT  EXPLORATION^^© 
DEFINITION  _(CEAB) 


3.0.1  Milestone  O-Conceot  Studies  Approval.  The  Milestone  (MS)  0  decision 
begins  Phase  0  in  which  alternative  EW  concepts  are  to  he  explored  to  identify  the  most 
promising  potential  solution(s)  to  validated  user  needs.  T&E  concerns  at  MS  0  are  data 
that  can  be  developed  during  Phase  0  that  support  a  MS  I  decision  with  respect  to  the 
alternative  concepts  and  the  development  of  recommended  T&E  "exit  criteria"  that  are 
established  and  presented  in  the  MS  0  Acquisition  Decision  Memorandum  (ADM). 

3.0.2  Phase  0-Concept  Explorationand  Definitjen. 

3.0. 2.1  Phase  Description.  Phase  0  typically  consists  of  competitive, 
parallel  short-term  concept  studies.  The  focus  of  these  efforts  is  to  define  and  evaluate 
the  feasibility  of  alternative  concents  and  to  provide  a  basis  for  assessing  the  relative 
merits  (i.  e.  advantages  and  disadvantages,  degree  of  risk)  of  these  concepts  at  the  next 
milestone  decision  point.  Analysis  of  alternatives  shall  be  used  as  appropriate  to 
facilitate  comparisons  of  alternative  concepts.  The  most  promising  system  concepts 
shall  he  defined  in  terms  of  initial,  broad  objectives  for  cost,  schedule,  performance, 
software  requirements,  opportunities  for  tradeoffs,  overall  acquisition  strategy,  and  test 
and  evaluation  strategy. 

3. 0.2.2  T&E.  The  main  objective  of  T&E  in  this  phase  is  to  assist  in 
defining  and  selecting  a  preferred  EW  System  concept,  candidate  technologies,  and 
critical  operational  characteristics. 

3.0.2.2.1  Developmental  Test  and  Evaluation.  DT&E  comprises  the 
major  test  events  and  evaluation  actions  within  this  phase.  The  primary  purpose  of 
DT&E  at  this  point  is  to  determine  whether  the  proposed  EW  concepts  are  feasible. 
Alternative  concepts  are  analyzed  against  required  key  technical  and  operational 
performance  parameters,  e.g.,  is  nanosecond  response  time  required  or  will 
microsecond  response  time  suffice?  Test  resources  and  tools  required  to  support  test 
and  evaluation  of  any  new  EW  system  that  may  be  developed  are  identified.  At  this 
juncture  within  most  EW  programs,  because  there  is  usually  little  EW  hardware  to  test 
at  this  stage,  much  of  the  smdies,  analyses,  concept  comparisons,  design  work,  and 
performance  appraisals  are  done  through  the  use  of  modeling  and  simulation.  In  some 
cases,  however,  as  a  result  of  an  Advanced  Technology  Demonstration  (ATD) 
program,  a  brassboard  or  breadboard  may  be  available  for  testing.  Considerable  testing 
of  concepts  and  some  testing  of  subsystems  occurs  in  this  phase. 

3.0.2.2.2  Operational  Test  and  Evaluation.  Significant  OT&E  will  not 
likely  occur  during  this  phase.  The  emphasis  in  this  phase  is  in  proving  out  the 
particular  concept  and  developing  cost  effective  alternatives  that  meet  the  user's  needs. 
To  that  end,  the  Operational  Test  Agency  (OTA)  reviews  program  documentation  and 
develops  a  working  relationship  with  the  user  in  order  to  understand  the  operational 
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requirements  and  define  the  critical  issues.  The  OTA  will,  as  a  minimum,  prepare  an 
initial  OT&E  outline  and  participate  in  acquisition  planning  to  ensure  that  schedules  and 
long-lead  items  meet  OT&E  objectives.  The  OTA  may  be  required  to  prepare  an  early 
operational  assessment  (EOA)  of  critical  systems,  subsystems,  and  components  may  be 
necessary  to  ascertain  that  risk  and  operating  environment  are  within  acceptable  limits. 

3.1  STEP  ONE.  Identify  Information  Needed  by  Decision  Makers  from  T&E. 

As  a  minimum,  T&E  addresses  the  (generic)  questions  as  depicted  in  Figure  10  and, 
more  specifically  as  listed  below  as  they  might  relate  to  each  EW  concept  or  concept 
comparison: 


(1)  What  T&E  data 
exist,  e.g.,  are  there 
historical  test  data 
available  on  previous 
EW  concepts,  jammer 
systems,  ECM 
techniques,  etc.  that 
would  preclude 
"reinventing  the 
wheel"?  Does 
preliminary  analysis 
conclude  that  the 
concept(s)  will  work? 

What  is  the 
confidence  level  in 
this  assessment?  Do 
the  available 

historical  data  on  the  testing  of  platform  self-protection,  for  example,  enhance 
the  confidence  that  an  evolutionary,  rather  than  revolutionary,  approach  is 
preferred? 

(2)  Can  requirements,  as  stated,  be  evaluated? 

(3)  Does  the  existing  EW  T&E  infrastructure/technology  base  permit 
evaluation?  Approximately  what  is  the  cost  and  time  frame?  If  not, 
approximately  what  is  the  cost  and  time  frame  to  create  the  infrastructure  and 
technology  base  to  adequately  assess  the  EW  concept(s)?  Does  the  existing 
technology  base  support  concurrent,  rather  than  sequential,  testing  of  parallel 
technologies? 

(4)  What  are  concept(s)  capabilities  and  limitations  versus  threats,  e.g.  does  the 
ECM  concept  adequately  address  the  monopulse  radar  threat? 

(5)  How  can  T&E  favorably  impact  risk  management  or  risk  reduction? 


Phase  0  -  Concept  Exploration  and  DeHnition 

For  each  concept, 

(1)  What  T&E  data  exist?  Does  analysis  conclude  that  the  concept(s)  will 
work?  What  is  the  confidence  level  in  this  assessment? 

(2)  Can  requirements,  as  stated,  be  evaluated? 

(3)  Does  the  existing  T&E  infrastructure/technology  base  permit  evaluation? 
Approximately  what  is  the  cost  and  time  frame?  If  not,  approximately  what  is 
the  cost  and  time  frame  to  create  the  infrastructure/technology  base? 

(4)  What  are  concept(s)  capabilities/limitations  versus  threats? 

(5)  How  can  T&E  favorably  impact  risk  management  or  risk  reduction? 

(6)  What  alternatives  exist  to  testing/test  assets?  What  are  the  risks? 

(7)  Has  the  preliminary  TEMP  been  approved?  Are  the  critical  technical 
parameters,  the  minimum  acceptable  operational  requirements,  and  the  critical 
system  characteristics  included? 

(8)  Is  each  proposed  concept  feasible?  Is  one  concept  preferred? 

(9)  What  are  the  recommended  Phase  I  T&E  "exit  criteria"? 

(10)  Does  the  Early  Operational  Assessment  (EOA)  address  the  availability  of 
and  planning  for  resources  projected  for  OT&E  test  events. 

Figure  10.  Phase  0  T&E  Questions 
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(6)  What  alternatives  exist  to  testing/test  assets?  What  are  the  risks  associated 
with  using  the  alternatives? 

(7)  Has  the  preliminary  TEMP  been  approved?  Are  the  critical  technical 
parameters,  the  minimum  acceptable  operational  requirements,  and  the  critical 
system  characteristics  included? 

(8)  Is  each  proposed  EW  concept  feasible?  Is  one  concept  preferred,  e.g.,  does 
an  onboard  EW  system  provide  more  or  less  survivability  of  the  host  platform 
than  an  integrated  onboard/offboard  system? 

(9)  What  are  the  recommended  Phase  I  T&E  "exit  criteria"? 

(10)  If  an  EOA  is  initiated  by  the  OTA  at  this  stage,  does  it  address  resource 
availability  and  planning  for  projected  OT&E  test  events? 

3.2  STEP  TWO.  Pre-test  Analysis.  Pre-test  Analysis  in  EW  CE&©  places 
maximum  emphasis  on  the  development  of  expected  outcomes  after  determining  the 
major  types  of  tests  needed,  i.e.,  open-loop  or  closed  loop,  HITL,  integration 
laboratory,  flying  testbed,  open  air  or  range.  Among  the  analytical  tools  developed 
during  pre-test  analysis  are  digital  models  and  simulations  of  elements  of  contending 
EW  concepts  that  are  sufficiently  representative  of  the  SUT  to  permit  conceptual  design 
trade-off  studies.  Conducting  pre-test  analysis  assists  in  structuring  test  conditions  prior 
to  the  conduct  of  test  events.  The  expected  response  of  the  SUT  can  be  used  to  design 
test  trials  to  evaluate  system  effectiveness  and  suitability.  For  the  initial  test  set-up,  test 
requirements  are  derived  from  the  system  development  and  engineering  activities. 

3.3  STEP  THREE.  Test  Activity  and  Data  Management.  This  step  emphasizes  a 
testing  approach  to  prove  out  the  feasibility  of  alternative  EW  concepts.  Test  planning 
involves  preparation  of  the  acquisition  program  baseline  (APB),  the  preliminary  TEMP 
and  an  initial  OT&E  outline.  The  actual  tests  should  be  structured  to  provide  data  for 
evaluation  of  acquisition  risk  and  decision  making.  Testing  in  the  CE&B  phase  is 
conducted  to  determine  whether  the  alternative  EW  concepts  can  meet  the  operational 
need,  e.g.,  can  the  missile  warning  system  meet,  or  exceed,  the  capability  to 
discriminate  among  at  least  the  top  10  (in  priority)  threat  missiles  against  which  it  is 
designed  to  operate.  Alternative  concepts  are  tested  against  the  requirements  outlined  in 
the  ORD  and  APB,  and  listed  in  the  RCM  for  tracking.  Test  events  are  conducted  at 
facilities  commensurate  with  EW  T&E  capabilities  required  for  that  event.  In  the 
CE&D  Phase  the  following  may  be  considered. 

Historical  Test  Data.  The  initial  step  in  any  EW  test  activity  should  be  the 
examination  of  previous  test  data  (or  descriptive  summaries)  stored  in  historical 
archives  to  ascertain  the  utility  of  previous  test  results,  i.e.,  does  the  required  new  EW 
capability  share  sufficient  commonality  with  any  predecessor  system  such  that  some 
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testing  can  be  obviated.  Development  planning  agencies,  Service  analysis  agencies  (like 
the  Air  Force  Center  for  Technology),  and  hardware  contractors  should  be  consulted 
when  researching  available  system  or  subsystem  historical  data  bases  or  data  bases  that 
should  share  data  at  military  laboratories.  The  use  of  TECNET,  INTERNET, 
Worldwide  WEB,  and  USENET  are  suggested  for  access  to  repositories  of  EW  data. 

A  thorough  search  of  available  test  data  sources  could  reduce  the  cost  of  the  T&E 
program  planned  for  the  potential  new  system.  Further,  modeling  and  simulation  of 
historical  test  data,  in  lieu  of  actual  testing  of  the  current  SUT,  should  be  performed  to 
fill  historical  information  voids.  This  is  a  highly  cost  effective  procedure  that  leads  to 
the  determination  of  what  data  are  lacking  and  needed  from  new  test  and  evaluation 
events. 


Component  measurement  test  events  are  less  likely  to  occur  during  this 
phase  than  in  subsequent  phases.  However,  in  some  cases  actual  equipment  in 
brassboard  or  prototype  configurations  may  be  available  for  measurement  test  events. 

Integration  testing  may  be  used  to  verify  fairly  complex  concepts  that 
require  sensor  fusion  (to  perform  a  more  accurate  Electronic  Support  function  such  as 
precision  emitter  location,  for  example)  or  that  have  operator  displays  and  operator 
actions  that  need  to  be  evaluated  with  real  time,  man-in-the-loop  simulations.  Emphasis 
should  be  on  lessons  learned  and  their  integration  with  data  available  from  computer 
simulations.  Test  efforts  should  make  efficient  use  of  computer  simulation  such  that 
sufficient  pre-test  analysis  can  be  accomplished  on  those  results  expected  from  the 
actual  laboratory  testing.  This  allows  comparison  of  measured  results  with  expected 
outcomes  to  validate  desired  test  conditions. 

Hardware-in-the-Loop  testing  is  important  in  this  phase  in  determining  the 
feasibility  of  each  EW  concept  under  test  and  in  evaluating  its  military  worth.  That  is, 
what  is  the  value  added  by  the  implementation  of  the  EW  concept?  For  concepts  that 
have  matured  to  a  hardware  stage,  HITL  test  events  can  be  used  to  refine  designs, 
check  system  performance  in  operational  environments,  and  work  out  early  problems 
by  using  either  flexible,  generic  EW  simulators/stimulators  or  commercial  laboratory 
equipment,  such  as  oscilloscopes,  synthesizers,  spectrum  analyzers,  signal  generators, 
etc. 


Installed  System  Test  (1ST)  Events  may  not  be  conducted  because  few 
concepts  will  have  reached  the  installation  stage  by  this  Phase.  If  possible,  however, 
1ST  events  are  used  to  look  for  optimum  solutions  to  stated  needs. 

Field/Open  Air  Tests  may  or  may  not  be  conducted  during  this  phase 
depending  on  the  availability  of  prototype  hardware.  If,  however,  advanced  technology 
development  (ATD)  programs  have  produced  brassboard  hardware,  it  may  be  installed 
in  testbeds  to  conduct  field  testing.  Data  may  be  obtained  in  this  manner  to  check 
equipment  operation  and  help  verify  non-real-time  computer  simulations  from  the 
CE&D  phase. 


Sirmilafinn/Stirnulation  Events.  Simulation  and  stimulation  events  are  used 
extensively  in  the  DoD  test  process.  They  can  be  applied  to  computer  or  physical 
working  models  or  the  SUT.  They  may  be  real  time  or  non-real  time  models.  Effective 
use  of  credible  models  and  their  simulation/stimulation  events  provides  cost  effective 
T&E.  Candidate  system  concepts  to  provide  the  needed  mission  capability  can  be 
analyzed  using  computer  simulations  of  mission  scenarios  integrated  with  models  of  the 
system  concepts.  The  use  of  ALARM  (described  in  paragraph  2.4.2.2.7.1),  or  a  similar 
model,  during  this  phase  should  be  considered.  ALARM  is  appropriate  for  assessing 
technical  characteristics. 

Data  management  occurs  during  this  phase.  In  this  step,  the  data  on  each 
EW  concept  are  converted  from  raw  data  collected  by  the  testers  into  measured 
outcomes  which  are  analyzed  for  validity  and  then  authenticated. 

3.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation  directly  identifies  the 
performance  capabilities  of  the  EW  system.  In  the  CE&©  phase,  the  alternatives 
concepts  identified  in  the  analysis  of  alternatives€9EA-are  judged  according  to  their 
merit  in  meeting  military  needs  as  stated  in  the  MNS  and  users'  requirements  as  stated 
in  the  ORD.  Assessments  identify  the  MOPs  for  the  systems  as  well  as  any 
shortcomings/deficiencies.  The  measured  outcomes  of  the  tests  are  compared  to  the 
expected  outcomes  established  in  Step  Two.  In  this  phase  the  greatest  disparity 
between  expected  and  measured  outcomes  occurs.  Measured  outcomes  of  testing,  when 
coupled  with  expected  results  from  computer  simulation,  provide  the  added  benefit  of 
refining  the  simulations  and  models  and  updating  their  parameters  in  order  to  better 
validate  the  model's  accuracy.  At  the  conclusion  of  the  post  test  synthesis  and 
evaluation  step,  all  test  objectives  will  have  been  analyzed  and  demonstrated  values  will 
have  been  documented.  The  final  output  of  this  step  must  be  T&E  information  for  the 
decision  makers  concerning  the  performance  parameters,  the  feasibility  of  each 
alternative  EW  concept,  the  availability  of  EW  test  resources,  and  the  adequacy  of  the 
technology  base  to  support  the  desired  concept. 

3.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 
Program  Information.  The  output  of  this  step  is  an  informed  decision  on  the  proper 
course  of  action  to  be  taken  at  this  point  predicated  upon  the  information  received  by 
the  decision  maker  from  the  application  of  the  Process  to  date. 

3.5.1  Milestone  I  Decision  Point.  The  results  of  the  Concept  Exploration -Md 
DefinitieflPhase  are  reviewed  and  approved  by  the  Milestone  Decision  Authority 
(MDA).  During  this  review,  the  MDA  confirms  that  study  efforts  support  the  need  for 
a  new  program,  that  the  system  concept(s)  shows  evidence  of  satisfying  the 
requirements,  that  the  threat  assessment  has  been  validated  as  required,  that  the 
proposed  EW  concept  is  producible  and  affordable,  that  risks  are  manageable,  and  that 
adequate  resources,  including  T&E  resources,"  can  be  programmed  to  support  an 
acquisition  program. 
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The  ADM  documents  the  decision  to  initiate  an  acquisition  program  and  enter  into 
Phase  B.  Program  Definition  and  Risk  Reduction  (DEMVAL).  It  also  approves  the 
Concept  Baseline  that  was  established  through  the  analysis  of  the  alterative  EW  system 
concepts.  Program-specific  exit  criteria  that  must  be  met  during  the  DEMVAL  Phase  I 
are  established. 

— 3.5.2  Documentation.  The  MPA  mav  require  anv  or  all  of  the  following 
documents  be  prenared/updated  at  Milestone  Ifollowing  describes-the-nrogrom 
documentation  required  by  DoD  Instruction  5000:2:  1)  The  STAR  is  the  primary  threat 
document  from  which  critical  intelligence  parameters  are  determined.  It  is  initially 
prepared  and  approved  by  the  appropriate  Intelligence  Agency  and  validated  by  the 
Defense  Intelligence  Agency  (DIA);  2)  The  ORD  is  initially  prepared  in  this  phase  by 
the  user  or  the  user's  representative.  The  ORD  reflects  the  critical  operational  issues 
(COIs)  and  their  criteria  from  which  the  minimum  acceptable  operational  performance 
requirements  are  derived;  3)  An  analysis  of  alternatives  mav  be  initially  prepared  at  MS 
0  and  updated  at  MS  I.  Significant  here  are  anv  changes  in  performance  parameters  or 
anv  adjustment  of  concepts. (Ed  sec  S)  bclowl  The  IPS  is  prepared  alMS-  I-r-It 
summarizes  the  results  of  Phase  0.  It  shall  identify  the  most  promising- concept(8)  to  be 
carried  into  Phase  I  and  the  reasons  for  elimination  of  alterative  concepts;  4) 

A  preliminary  TEMP  containing  the  requirements  for  T&E,  the  initial  management 
responsibilities,  an  outline  of  DT&E  and  OT&E,  and  resource  requirements  must  be -is 
prepared.  This  is  the  primary  T&E  document  for  review  authorities.;  and  5)  A  COE  A 
is  initially  prepared  at  MS  0  and  updated-at  MS  I.  Significant  here  is  any  change  in 
performance  parameters  or  any  adjustment  of  conceptsr 


4.  EW  SYSTEMS  T&E  DURING  PHASE  I  -PROGRAM  DEFINITION  AND 
RISK  REDUCTION  DEMONSTRATION  AND  VALIDATION  (DEM/-VAfc) 


4.0.1  Milestone  I-Concept  Demonstration  Approval.  The  MS  I  decision, 
rendered  in  an  ADM,  determines  if  the  results  of  Phase  0  warrant  establishing  a  new 
acquisition  program,  selects  the  preferred  EW  concept(s)  to  continue  development,  and 
approves  entry  into  Phase  I.  This  concept(s)  will  evolve  into  engineering  prototypes 
(EPs)  for  continued  T&E.  The  T&E  outcome  is  the  assessment  of  each  EP. 

4.0.2  EW  Systems  in  Phase 


4.0.2. 1  Phase  Description.  During  this  phase,  the  program  becomes  defined 


as  one  or  more  concepts,  design  approaches,  and/or  parallel  technologies  are  pursued  as 


warranted.  Assessments  of  the  advantages  and  disadvantages  of  alternative  concepts  are 


refined.  Prototyping,  demonstrations,  and  early  operational  assessments  are  considered 


and  included  as  necessary  to  reduce  risk  so  that  technology,  manufacturing,  and  support 


risks  are  well  in  hand  before  the  next  decision  point.  Cost  drivers,  life-cycle  cost 


estimates,  cost-perfoimance  trades,  interoperability,  and  acquisition  strategy  alternatives 
are  considered  to  include  evolutionary  and  incremental  software  development 


4.0. 2.2  T&E  in  Phase  I DEM/VAE.  Because  Phase  IDEM/VAL  occurs 
early  in  the  EW  T&E  process,  tests  and  evaluations  may  be  conducted  to  validate  and 
qualify  the  design  and  to  ensure  that  the  product  is  ready  for  Government  acceptance. 
Special  emphasis  should  be  placed  on  the  use  of  HITL  test  events  to  test  systems 
rigorously  in  a  design  exploration  and/or  refinement  effort  prior  to  Government 
validation  testing. 

4.0.2.2.1  Developmental  Test  and  Evaluation.  DT&E  conducted  during 
Phase  T  the  DEM/VAL  phase  is  used  to  demonstrate:  1)  that  technical  risk  areas  have 
been  identified  and  reduced  to  acceptable  levels;  and  2)  that  the  best  technical 
approaches  have  been  adopted  accepted:  and  3)  that  from  this  point  on. -engineering 
efforts,  rather  than  experimental  efforts,  are  required.  DT&E  supports  the  MS  II 
decision  which  considers  entry  into  Full-Scale  Development  and,  as  appropriate,  low 
rate  initial  production  (LRIP).  An  EP  can  be  built  in  this  Phase  by  replacing  certain 
components  in  the  DSM  with  hardware  components  developed  during  the  phase  .DEM/ 

4.0.2.2.2  Operational  Test  and  Evaluation.  OT&E  for  EW  systems  during 
Phase  Itho  DEM/VAL  phase  is  conducted  to  support  the  MS-II  decision  regarding  a 
system's  readiness  to  move  into  Engineering  and  ManufacturingFull  Scale 
Development.  As  part  of  OT&E  planning,  the  operational  aspects  of  the  proposed 
technical  approach  are  examined  by  the  OTA.  Consistent  with  the  evolutionary 
requirements  definition,  the  OTA  works  with  the  user  to  refine  proposed  performance 
objectives  and  identify  surge  and  mobilization  requirements.  Also,  evaluation  criteria 
should  be  documented  and  the  test  schedule  reviewed  for  adequacy  of  time  and  that  the 
availability  of  test  articles  is  sufficient  to  meet  the  OT&E  objectives.  This  may  include 


265226 


provisions  for  an  Early  Operational  Assessment  (EOA).  If  the  MS  II  decision  includes 
an  LRIP,  an  EOA  is  required. 

4.1  STEP  ONE.  Identincation  of  T&E  Information  Required  by  Decision 
Makers  from  T&E.  For  Phase  Ithe  DEM/VAL  phase,  the  test  and  evaluation 
requirements  would  have  been  identified,  updated,  and  carried  forward  from  the  CE&D 
phase.  The  TEMP  is  updated  and  prepared  for  approval  at  MS  H.  If  required,  an  EOA 
is  prepared.  The  ROM  and  the  SMM  are  updated.  Exit  criteria  for  this  phase  are  also 
defined. 

4.1.1  As  a  minimum,  T&E  addresses  the  (generic)  questions  as  depicted  in 
Figure  11  and,  more  specifically,  as  listed  below  as  they  might  relate  to  each  EW  EP: 


(1)  Existing  data  analysis  -  will 
the  EP(s)  work?  What  is  the 
confidence  level  in  this 
assessment?  To  what  extent,  for 
example,  can  the  situational 
awareness  be  enhanced  by  an  RWR 
EP,  i.e.,  does  it  give  the  operator 
full  four  quadrant  coverage? 

(2)  Can  the  requirements,  as 
stated,  be  evaluated? 

(3)  Does  the  existing  T&E 
infrastructure  and  technology  base 
permit  evaluation?  Approximately 
what  is  the  cost  and  time  frame?  If 
not,  approximately  what  is  the  cost 
and  time  frame  to  create  the 
appropriate  infrastructure  and 
technology  base?  If  this  is  a  software-controlled  EW  EP,  are  there  software 
modifications  to  be  addressed? 

(4)  What  are  the  capabilities/  limitations  of  each  EW  EP  versus  threats?  Is  the 
receiver  sensitivity  sufficient  to  detect  low-probability-of-intercept  emitters?  Is 
the  circuitry  sufficiently  hardened  against  high  power  microwaves?  Should  it  be? 
Is  the  threat  library  current? 

(5)  How  can  T&E  favorably  impact  risk  management  or  risk  reduction?  Is  there 
a  moderate-to-high  risk  technology  issue  that  can  be  resolved  early  in  this 
phase? 

(6)  What  alternatives  exist  to  testing/test  assets?  What  are  the  risks? 


Phase  I  -  Program  Definition  and  Risk  Reduction 

For  each  engineering  prototype  (EP), 

(1)  Existing  data  analysis  -  will  the  EP(s)  work?  What  is  the  confidence 
level  in  this  assessment? 

(2)  Can  the  requirements,  as  stated,  be  evaluated? 

(3)  Does  the  existing  T&E  infrastmcture/technology  base  permit 
evaluation?  Approximately  what  is  the  cost  and  time  frame?  If  not, 
approximately  what  is  the  cost  and  time  frame  to  create  the 
infrastructure/technology  base? 

(4)  What  are  the  capabilities/limitations  of  each  EP  versus  threats? 

(5)  How  can  T&E  favorably  impact  risk  management  or  risk  reduction? 

(6)  What  alternatives  exist  to  testing/test  assets?  What  are  the  risks? 

(7)  Has  TEMP  been  approved?  The  TEMP  must  contain  the  performance 
parameters  reflected  in  the  ORD,  analysis  of  alternatives,  and  APB. 

They  must  be  consistent. 

(8)  Is  each  proposed  EP  feasible?  Is  one  preferred? 

(9)  What  are  the  recommended  Phase  II  T&E  "exit  criteria"? 

(10)  What  are  the  recommended  criteria  for  certification  of  readiness  for 
final  OT&E? 

(11)  Does  the  EOA  address  the  early  projection  of  potential  operational 
effectiveness  and  suitability  criteria? 


Figure  1 1 .  Phase  I  T&E  Questions 
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(7)  Has  TEMP  been  approved?  The  TEMP  must  contain  the  performance 
parameters  reflected  in  the  ORD,  analysis  of  alternatives  -GQEA.  and  APB. 

They  must  be  consistent. 

(8)  Is  each  proposed  EP  feasible?  Is  one  preferred,  and  if  so,  why? 

(9)  What  are  the  recommended  Phase  II  T&E  "exit  criteria"? 

(10)  What  are  the  recommended  criteria  for  certification  of  readiness  for  final 
OT&E? 

(11)  Does  the  EOA  address  the  early  projection  of  potential  operational 
effectiveness  and  suitability  criteria? 

4.1.2  These  questions  are  basically  identical  to  the  Phase  0  questions.  However, 
because  the  program  is  more  mature,  more  data  at  higher  confidence  levels  will  exist. 

4.2  STEP  TWO.  Pre-test  Analysis.  Because  Pre-test  Analysis  is  more  concerned 
with  the  "how  to",  or  the  methodology,  of  testing,  the  Pre-test  analysis  Step  in -fee 
DEM/VAL  phase  Phase  I  seeks  to  build  on  the  DT&E  actions  from  Phase  0  by  refining 
the  test  trials  and  test  conditions  to  bring  them  closer  to  operational  reality.  The 
correlation  between  the  model  and  the  EP  should  be  closer;  and,  both  should  be 
converging  toward  a  match  with  the  operational  requirement.  Development  of  expected 
outcomes  is  again  emphasized.  Future  testing  will  again  be  able  to  take  advantage  of  the 
data  accumulated  during  this  phase. 

4.3  STEP  THREE.  Test  Activity  and  Data  Management.  The  objective  of  the 
T&E  Process  during  Phase  I  DEM/VAL  is  to  gather  data  to  make  trade-offs  and  to 
identify  the  preferred  technical  approach  for  satisfying  an  operational  requirement.  In 
the  DEM/VALPhase  I  T&E.  an  EP  should  be  tested  to  determine  the  expected 
performance  of  the  proposed  EW  system.  Test  program  planning  should  also  identify 
any  required  upgrades  to  existing  test  resources  or  the  need  for  new  test  resources  in 
subsequent  phases  of  the  program  which  were  not  identified  at  MS  I.  Wherever 
practical,  testing  is  to  be  planned  and  conducted  to  take  full,  cost  effective  advantage  of 
existing  investments  in  DoD  ranges,  facilities,  and  other  resources,  unless  otherwise 
documented  in  the  TEMP.  In  DEM-Phase  I  the  following  are  considered. 

Historical  Test  Data.  Building  on  the  historical  data  search  and  evaluations 
performed  during  Phase  0,  historical  data  in  Phase  I  may  be  extremely  valuable  in 
developing  confidence  that  the  proposed  development  is  possible  and  in  reducing 
development  risk  by  identifying  successes  and  failures  in  earlier  programs. 

Measurement  Tests.  Prototype  hardware  may  be  subjected  to  the  measurement  of 
system  parameters  such  as  antenna  gain  patterns,  receiver  sensitivity,  ERP,  etc.  This 


type  of  data  is  needed  as  input  to  the  analysis  of  system  operation  and  to  computer 
simulations.  Representative  measurements  that  are  typically  accomplished  during  Phase 
I  include: 

a.  The  directional  accuracy  of  an  antenna,  its  beamwidth,  and  gain  pattern. 

b.  The  radar  cross  section  (RCS)  of  prototype  lethal  suppression  missiles. 

c.  The  IR  signamres  of  prototype  lethal  suppression  missile  engines. 

I  Integration  Tests.  It  is  likely  that  Phase  I DEM/VAL  testing  will  commence 
before  components  are  tested  and  evaluated  by  the  Government.  This  testing  should 
focus  on  identifying  EW  system  hardware  and  software  problems,  maturing  system 
performance,  and  evaluating  estimated  reliability,  availability,  and  maintainability 
(RAM)  levels.  The  following  are  typical  tests  that  can  be  conducted  at  this  stage  of  the 
process:  receiver  stimulation,  database  characterization,  display  compatibility,  and 
fault  testing. 

Hardware-in-the-Loon  Tests.  When  system  integration  laboratory  testing  has  been 
completed,  engineering  prototype  components  should  be  ready  for  HITL  testing.  Initial 
testing  should  be  conducted  against  a  rigorous  test  environment  and  mature  the  EW 
system  in  a  non-adversarial  manner.  The  main  thrust  of  HITL  testing  is  to  evaluate  the 

1  performance  of  actual  hardware  systems/subsystemsfind  in  simulated  environments. 

I  Installed  System  Tests  on  final  platforms  will  usually  not  occur  in  this  ^hase  of 

system  development.  However,  testing  could  be  used  for  equipment  mounted  on  test 
platforms  where  security  and/or  signal  density  requirements  can  best  be  accommodated. 

Open- Air /Field  Tests.  The  effects  of  some  real-world  environmental  conditions 
can  be  investigated  only  under  actual  exposure  to  those  conditions.  Therefore,  host 
platform  testing  may  be  required  early  in  an  EW  system's  development  or  modification 

I  cycle  in  order  to  evaluate  its  achieved  performance.  During  -DEMPhase  1.  this  can  be 
accomplished  by  installing  the  EW  system  engineering  prototype  in  a  spread  bench 
configuration,  aboard  a  large-body  testbed  aircraft  (or  on  a  highly  instrumented  rocket 
sled,  when  precision  position  and  velocity  data  are  of  interest),  or  on  a  mobile  or  fixed 
surface  platform  that  would  be  indicative  of  the  "real-world"  environment. 

I  Simulations/Stimulation  Events.  As  in  the  CE&D  Phase,  modeling  and 

simulation  should  be  used  in  lieu  of  actual  testing  where  feasible  to  fill  data  voids. 
Hardware-in-the-loop  facilities  and  open-air/field  testing  events  provide  a  capability  to 
test  against  man-in-the-loop  threat  simulations  and  surrogate  weapon  systems.  The 
number  of  people  involved  and  the  complexity  of  the  simulators  can  make  this  testing 
more  expensive  than  in-plant  system  integration  laboratory  testing.  Extensive  pre-test 
analysis  using  computer  simulations  should  be  conducted  to  obtain  the  maximum  benefit 
from  subsequent  hardware-in-the-loop  and  open-air/field  testing.  The  use  of  a  model 


like  ALARM,  or  a  similar  model,  to  assess  technical  performance  should  be 
considered. 

Data  Mana>yement.  Data  from  the  test  events  and  trials  in  the  DEM/  Phase  I  are 
processed,  as  in  the  CEife©  phase,  to  convert  raw  data  to  a  form  that  is  understandable 
and  useable  to  decision  makers. 

4.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation  consists  of  reducing  and 
correlating  these  measured  data  for  technical  performance  parameters,  measures  of 
effectiveness  (MOEs),  and  measures  of  performance  (MOPs),  on  both  a  quick-look  and 
a  thorough  basis.  This  iterative  assessment  process  begins  with  measurement  testing 
and  continues  through  die-open  air  testing  conducted  during  DEM/VAL.  The  expected 
outcomes  from  Step  Two  should  be  used  in  computer  simulations  to  make  early 
assessments  of  operational  effectiveness  and  suitability  of  the  EP.  Data  collected  from 
both  the  EW  SUT  and  threat  systems  are  correlated  with  aircraft  flight  paths  as  a 
function  of  time.  Instrumentation  that  monitors  the  electromagnetic  environment 
collects  jammer  and  radar  signal  parameters  and  measures  power  levels.  These  data  are 
compared  with  computer  simulation  or  engineering  estimations.  All  data  are  analyzed  to 
determine  test  results. 

4.4.1  Results  from  this  ^hase  must  be  reported  through  appropriate  channels  to 
the  MDA.  The  results  of  these  tests  and  evaluations  will  have  a  major  impact  on  future 
program  decisions.  Where  required,  OT&E  assessments  are  to  be  reported  at  the  end  of 
each  phase  of  T&E.  The  reports  from  OAs  and  EGAs  are  to  be  submitted  to  the  MDA 
or  PDA  as  required.  These  reports  should  identify  test  limitations  and  their  effects  on 
the  ability  to  demonstrate  whether  or  not  critical  systems  design  issues  and  risk  have 
been  resolved.  Results  may  lead  to  changes  in  specifications  or  may  result  in  a  redesign 
of  the  system. 

4.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 
Program  Information.  The  output  of  this  step  is  an  informed  decision  on  the  proper 
course  of  action  to  be  taken  at  this  point  predicated  upon  the  information  received  by 
the  decision  maker  from  the  application  of  the  Process  to  date. 

4.5.1  Milestone  11  Decision  Point.  During  the  Milestone  II  review,  the 

MD A/PDA  reviews  the  results  to  confirm  that  they  provide  reasonable  assurance  that 
the  technologies  and  processes  critical  to  successful  development  of  the  EW  system  are 
attainable  and  that  the  threat  assessment  and  mission  need  are  still  current  and  valid. 

The  MDA  approves  the  the  acquisition  strategy,  the  Cost  as  an  Independent  Variable 
tCAIV)  objectives.  tThe  LRIP  strategy  and  quantities,  the  exit  criteria  for  Phase  H,  -asd 
the  Developmental  Acquisition  Program  Bbaseline  (APB),  and  entry  are  all  approved; 
and  approval  is  granted  to  enter-into  Phase  II. 
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4.5.2  Developmental  Baseline.  Compared  to  the  Concept  Baseline,  the 
Developmental  Baseline  contains  more  detailed  and  refined  cost,  schedule,  and 
performance  objectives  and  thresholds.  The  APB  shall  contain  only  the  most  important 
cost,  schedule,  and  performance  parameters.  The  list  of  system  and  technical 
performance  parameters  should  be  expanded  from  the  key  parameters  of  the  Concept 
Baseline  that  were  derived  from  testing -the  engineering  prototype:-Performance 
objectives  set  for  Phase  II  should  represent  a  realistic,  meaningful,  measurable,  cost- 
effective,  and  affordable  increment  in  operational  capability  beyond  the  minimum 
acceptable  requirements. 

4.5.3  Documentation.  The  MPA  may  require  any  or  all  of  the  following 
documents  be  updated  at  Milestone  II  following  is  the  program  documentation  required 
by  DoDI  5000.2:  1)  The  STAR  is  updated  at  MS  II  and  at  other  points  in  the  program 
determined  by  the  MBA.  It  reflects  new,  validated  threat  assessments,  as  well  as 
addresses  any  unresolved  threat  concerns  and  critical  intelligence  parameters  that  have 
surfaced  during  Phase  I  DEM/VAL  T&E;  2)  The  ORD  4s-mav  be  updated  and 
expanded  for  MS  11  to  include  thresholds  and  objectives  for  more  detailed  and  refined 
performance  capabilities  and  characteristics  based  on  the  results  of  trade-off  studies  and 
testing  done  during  Phase  I4_t  Key  parameters  from  the  ORD  are  included  in  the 
Development  Baseline  at  MS  II;  3)  The  IPS  at  MS  11  summarizes  the  results  of  Phase-I 
and  how-fiio  exit  criteria  in  the  MS  I  ADM  were  satisfied.  It  also  identifies  further  risk 
reduction  offorta,  trade  off  decisions,  a  summary  of  cost  estimates  and  assessments,  and 
the  proposed  acquisition  strategy,  proposed  waivers,  any  LRIP  quantities  and  the  T&E 
events  to  be  accompliahed  prior  to  LRIP  contract  award;  4)  The  TEMP  is  finalized  and 
contains  data  necessary  to  conduct  DT&E  and  OT&E.  DT&E  events  follow  the  conduct 
of  tost  events  and  are  labeled  asT^TI,  DTIA,  DTIB,  etc.  OT&E  events  are  labeled  in  a 
similar  maimer.  The  update  should  describe  testing  performed  to  date  and-the 
nttninment  of  thresholds  and  objectives  of  performance  parameters.  The  SMM  and  the 
RCM  should  be  ufe-updated:  and  45)  the  analysis  of  alternatives  -GOEA-at  MS  II 
addresses  the  most  promising  system  concept  demonstrated  and  validated  in  Phase  II. 
Performance  and  cost  intervals  should  be  narrowed  to  point  estimates.  A  sensitivity 
analysis  should  be  is  performed  to  identify  any  critical  sensitivities  of  the  EW  system's 
effectiveness  to  test  restrictions,  such  as  safety  constraints  or  test  resource  limitations. 
The  TEMP  must  be  finalized  prior  to  Milestone  II.  The  update  describes  DT&E  and 
OT&E  testing  performed  to  date,  the  attainment  of  thresholds  and  objectives  of 
performance  parameters,  and  remaining  DT&E  and  OT&E  necessary  to  meet  Milestone 
III  exit  criteria. 
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5.  EW  SYSTEMS  T&E  DURING  PHASE  II  -  ENGINEERING  AND 
MANUFACTURING  DEVELOPMENT  (EMD) 


5.0.1  Milestone  IT  Developmental  Approval  to  Enter  Engineering  and 
Manufacturing  Development.  The  MS  II  decision  determines  if  the  progress  of  the 
selected  EP(s)  warrant  fer-continued  development  as  production  prototypes  (PPs).  The 
T&E  outcome  is  the  assessment  of  each  selected  prototype. 

5.0.2  EW  Systems  in  Phase  II  -  EMD. 

5.0.2. 1  Phase  Description.  The  primary  objectives  of  this  phase  are  to:  translate 
the  most  promising  design  approach  into  a  stable,  interoperable,  producible,  supportable, 
and  cost-effective  design:  validate  the  manufacturing  or  production  process:  and,  assess 
system  capabilities  through  test  and  evalaution.  Low  Rate  Initial  Production  (LRIPl 
occurs  while  the  EMD  nhase  is  still  continuing  as  test  results  and  design  fixes  or  upgrades 
are  incorporated.  The  main  objective  of  this  phase  is  to  design,  fabricate,  test  and 
evaluate  an  EW  PP;  to  validate  the  manufacturing  or  production' process  confirming 
that  design  risks  have  been  mitigated;  and  to  demonstrate  through  testing  that  the 
system  capabilities  meet  specificationrequirements  and  satisfy  mission  needs 

5. 0.2.2  T&E  in  EMD.  The  main  objectives  of  T&E  in  EMD  are  to  ensure  that 
engineering  is  complete,  to  validate  system  performance  of  the  production  design  in  a 
realistic  environment,  and  to  ensure  that  specifications  are  met  through  preproduction 
and  production  qualification  testing.  Further,  a  test  plan  is  developed  to:  ensure  design 
problems  are  solved;  provide  the  guidelines  for  software  testing;  validate  configuration 
changes;  validate  system  compatibility  and  interoperability  with  other  EW  systems 
and/or  other  platform  electronic  systems;  and  continue  the  reliability,  availability,  and 
maintainability  program. 

5. 0.2. 2.1  Developmental  Test  and  Evaluation.  DT&E  conducted  during  this 
phase  provides  the  final  technical  data  to  determine  the  EW  system's  readiness  to 
transition  into  either  LRIP  or  full-rate  production.  DT&E  in  EMD  is  conducted  with 
prototype  hardware  and  is  characterized  by  the  use  of  engineering  and  scientific 
approaches  under  controlled  conditions  to  provide  quantitative  and  qualitative  data  for 
use  in  the  system's  evaluation.  Technical  performance  is  measured  with  respect  to  both 
operational  effectiveness  and  suitability  factors  that  include  reliability,  availability, 
maintainability,  compatibility,  interoperability,  safety,  and  supportability.  The 
develoning  agency  hnnlomenting  command  must  certify  that  the  EW  system  is  ready  for 
lOT&E. 


5.0. 2.2. 2  Operational  Test  and  Evaluation.  The  primary  purpose  of  EMD 
OT&E  is  to  support  a  full-rate  production  decision  at  Milestone  III.  An  OA  may  be 
required  to  support  LRIP  of  the  system.  After  the  developing  agency  implementing 
command  certifies  that  the  system  is  ready  for  lOT&E,  the  OTA  conducts  lOT&E  on  a 
representative  production  system  to  evaluate  operational  effectiveness  and  suitability 
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tests  and  to  ensure  that  the  system  meets  its  minimum  operational  thresholds.  lOT&E 
also  addresses  logistics  and  software  support  requirements,  identifies  deficiencies  or  the 
need  for  modifications,  and  provides  information  to  refine  training,  tactics, 
countermeasure  techniques,  and  doctrine. 

5.1  STEP  ONE.  Identiflcation  of  T&E  Information  Required  By  Decision 
Makers  from  T&E.  The  main  objective  of  the  Process  in  EMD  is  to  ensure  that  the 
defined  and  selected  operational  effectiveness  and  suitability  performance  parameters, 
along  with  design,  specification,  and  production  characteristics,  are  still  valid  for  the 
EW  SUT. 

5.1.1  As  a  minimum,  T&E  addresses  the  (generic)  questions  as  depicted  in 
Figure  12  and,  more  specifically 
as  listed  below  as  they  might 
relate  to  each  EW  PP: 

(1)  For  DT&E,  what 
are  the  capabilities 
and  limitations  of 
each  PP  being 
developed?  What  is 
the  confidence  level 
in  these  data  and/or 
this  assessment? 

(2)  For  OT&E,  are 
the  PPs  suitable  and 

effective  in  satisfying  j2.  Phase  II  T&E  Questions 

the  mission  need? 

What  is  the  confidence  level  of  this  assessment?  Are  the  PPs  "user 
friendly",  i.e.,  are  typical  users  exposed  to  the  SUT  and  their  reactions  to 
system  operations  considered? 

(3)  Have  key  performance  objectives/thresholds  been  validated  versus 
advanced  threats?  Are  the  simulated  threats  sufficiently  representative  both 
in  numbers  and  capability? 

(4)  Are  the  PPs  feasible?  Do  they  satisfy  the  need?  Is  one  preferred?  Is  the 
design  stable  and  producible? 

(5)  Have  requirements  changes  been  incorporated  into  the  APB,  ORD,  and 
contract  specifications? 

(6)  Have  specification  changes  been  reflected  back  into  the  requirements 
and  incorporated  into  the  APB  and  ORD? 


Phase  II  -  Engineering  &  Manufacturing  Development 

Far  each  production  prototype  (PP), 

(1)  For  DT&E,  what  are  the  capabilities  and  limitations  of  each  PP  being 
developed?  What  is  the  cwifidence  level  in  this  data/assessment? 

(2)  For  OT&E,  are  the  PPs  suitable  and  effective  in  satisfying  the  mission 
need?  What  is  the  confidence  level  of  this  assessment? 

(3)  Have  key  performance  objectives/thresholds  been  validated  versus 
advanced  threats? 

(4)  Are  the  PPs  feasible?  Do  they  satisfy  the  need?  Is  one  preferred? 

(5)  Have  requirements  changes  been  incorporated  into  the  APB,  ORD  and 
contract  specifications? 

(6)  Have  specification  changes  been  reflected  back  to  requirements  and 
incorpOTated  into  the  APB  and  ORD? 

(7)  Does  the  TEMP  reflect  the  changes  in  (5)  and  (6)?  Has  it  been  approved? 


(7)  Does  the  TEMP  reflect  the  changes  in  (5)  and  (6)?  Has  it  been 
approved? 

5.2  STEP  TWO.  Pre-test  Analysis  is  conducted,  as  required,  to  analyze  test  trial 
conditions  or  establish  expected  outcomes  before  progressing  through  an  orderly  test 
process  beginning  with  system  integration  laboratory  (SIL)  T&E  and  ending  with 
field/open-air  range  testing.  Evaluation  actions  in  this  Phase  are  continuous  and  begin 
with  the  Pre-test  analysis.  Computer  aided  simulation  analysis  may  be  used  to  assess  the 
expected  performance  of  the  EW  system  under  various  test  conditions  and  to  complete 
any  remaining  pretest  planning  factors.  In  addition,  the  DSM,  if  used,  is  updated  to 
support  future  evaluation  of  engineering  change  proposals  (ECPs)  and  modifications. 

5.3  STEP  THREE.  Test  Activity  and  Data  Management.  EMD  T&E  is 
conducted  to  determine  whether  the  EW  system  meets  all  critical  operational 
requirements  when  it  is  produced  in  a  cost  effective  manner.  It  is  significant  that  the 
manufacturing  and  production  process  is  validated  at  this  point,  and  it  represents  a  final 
opportunity  for  modifications  to  be  made  to  the  system  design  before  the  EW  system 
enters  final  production.  T&E  conducted  is  commensurate  with  the  capabilities  required 
for  that  event.  In  the  EMD  Phase  the  following  are  considered. 

Historical  Test  Data.  During  EMD,  historical  data  searches  can  fulfill  several 
functions:  1)  Historical  data  may  obviate  the  need  for  certain  tests,  2)  It  may  identify 
previously  unsuccessful  designs  and  either  suggest  improvements  or  steer  the  program 
clear  of  pitfalls,  and  3)  It  may  suggest  or  provide  answers  to  questions  raised  when 
actual  performance  of  the  current  SUT  does  not  meet  expectations. 

Component  Measurement  Test  events  at  this  stage  are  much  more  likely  and 
useful.  With  the  EW  system  installed  on  projected  host  platforms,  valid  measurements 
can  be  made  which  confirm  design  capabilities,  identify  design  deficiencies,  and 
determine  employment  options.  Measurement  testing  should  establish  values  for 
technical  performance  parameters  for  installed  antenna  patterns,  platform  signatures, 
direction  finding,  component  reliability  under  various  field  tests,  and  decoy  trajectories. 

Integration  Tests.  Similar  to  Phase  I  DEM/VAE  testing,  the  EMD  testing  may  | 
begin  in  a  test-fix-test  sequence.  An  electromagnetic  environment  generator  can  be  used 
to  support  this  testing  by  stimulating  components  of  the  engineering  prototype  to 
evaluate  performance  and  compliance  with  technical  requirements.  This  testing  should 
confirm  that  performance  thresholds  capable  of  being  tested  in  the  integration 
laboratory  have  been  achieved  and  correct  any  identified  hardware  and  software 
problems.  Once  the  components  of  the  SUT  have  been  tested  and  evaluated  they  can  be 
prepared  for  more  rigorous  evaluation. 

Hardware-In-The-Loop  (HITL)  Tests.  The  production  prototype  components  are 
usually  tested  in  Government  HITL  test  facilities  before  the  system  is  installed  in  a 
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testbed  for  field/open-air  testing.  This  ground  testing  should  focus  on  confirming  that 
identified  problems  have  been  fixed,  that  performance  thresholds  can  be  achieved,  and 
on  optimizing  countermeasure  techniques  versus  HITL  threat  simulators.  Because  a 
complete  EW  system  is  available  for  this  testing  (instead  of  just  critical  components  as 
I  in  Phase  T  DEM/VAfc  testing),  this  is  the  first  opportunity  to  conduct  integrated  system 
effectiveness  tests.  Specific  tests  to  be  conducted  depend  on  the  functions  included  in 
the  system.  Some  examples  are: 

a.  Testing  the  capability  of  radio  frequency  (RE)  warning  receivers  to 
process  a  high-density  signal  environment. 

b.  Optimizing  RE  countermeasures  versus  manned  hybrid  threat 
simulators  and  demonstrating  achieved  effectiveness  in  terms  of  tracking  errors  and 
missile  miss  distances. 

c.  Determining  the  effectiveness  of  RF  countermeasures  versus  early 
waming/GCI  radars  and  communication  links. 

Installed  System  Tests  are  normally  the  first  opportunity  to  evaluate  system 
operation  on  a  weapon  system  platform.  They  are  conducted  to  evaluate  the  integrated 
performance  of  EW  subsystems  as  part  of  a  weapon  system  platform.  A  prime  purpose 
is  to  test  the  electromagnetic  compatibility  (EMC)  of  the  EW  SUT  with  other  systems 
on  the  host  platform.  Some  examples  are: 

a.  Stimulate  the  EW  system  with  mission  representative  threat  signal 
environment  and  verify  correct  display  when  communication,  navigation,  and 
identification  avionics  systems  are  transmitting. 

b.  Confirm  that  RF  receivers  can  detect  and  identify  threats  in  specified 
times  while  blanked  for  jammer  transmissions. 

c.  Measure  jammer  duty  cycle  to  ensure  it  meets  specifications. 

_ d.  Confinn  jamming  waveform  and  power  level  integrity  is  maintained 

throughout  the  system. 

Field/Open- Air  Tests  present  the  first  opportunity  to  measure  selected 
performance  parameters  of  the  EW  system  in  the  actual  operating  environment  of  its 
host  weapon  system  platform.  They  provide  the  means  to  calibrate  the  other  classes  of 
events  (i.e.,  digital  simulations,  system  integration,  HITL,  installed  system  tests)  and  to 
validate  the  expected  outcomes  and  the  measured  outcomes  thereby  establishing  an 
acceptable  confidence  factor.  Flight  testing  normally  begins  with  one-on-one  scenarios 
and  progresses  to  multiple  threat  system  scenarios  constructed  to  be  as  operationally 
realistic  as  possible,  using  available  threat  simulator  resources,  ranges,  and  range 
instrumentation.  Some  examples  of  EMD  open-air  range  testing  are: 


systems. 


a.  Conduct  pod  stores  release  certification  tests  for  pod-mounted 


b.  Confirm  that  threat  signals  are  detected  at  specified  ranges  and 
correctly  identified. 

c.  Determine  if  countermeasures  degradation  of  threat  performance 
parameters  meets  required  thresholds. 

For  IOT«&E,  field/open-air  testing  is  indispensable  for  evaluating  operational 
effectiveness  and  suitability  because  it  provides  the  final  basis  for  comparison  of 
previously  collected  data  and  a  point  of  departure  for  additional  simulation,  analysis, 
and  evaluation. 

■Sirrmlatinns/Stimulation  Events.  As  in  Phase  Itho  DEM/VAL  Phase,  modeling 
and  simulation  should  be  used  in  EMD  in  lieu  of  actual  testing  where  feasible  to  fill 
data  voids.  Computer  simulations  should  be  run  to  support  pre-test  analysis  in  order  to 
structure  new  test  trials  and  update  the  values  expected  for  system  performance  and 
technical  performance  parameters.  The  use  of  models  like  AASPEM  and 
SUPPRESSOR,  or  similar  models,  during  this  phase  should  be  considered.  Both 
AASPEM  and  SUPPRESSOR  are  appropriate  for  assessing  operational  performance. 

Data  Management.  Data  (e.g.,  antenna  patterns,  time  responses,  tracking, 
telemetry,  firing  events,  operator  logs)  measured  in  various  test  facilities  must  be 
processed  into  a  form  suitable  for  analysis.  The  actual  data  may  be  understandable  only 
to  the  analyst.  This  is  particularly  true  during  the  early  forms  of  testing  when  DSMs 
may  be  used  more  than  the  actual  hardware.  In  this  step,  the  data  are  mmed  into 
meaningful  information  which  can  be  understood  by  the  average  person  who  has  some 
familiarity  with  the  system  and  the  test  requirements.  Test  managers  must  work  closely 
with  the  test  force  and  test  range/facility  personnel  to  determine  what  data  are  collected 
and  the  required  reporting  formats  for  the  data.  Once  processed,  time  correlated 
measurement  values  can  be  used  to  conduct  a  post-test  assessment  of  the  results. 

5.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation.  As  higher-confidence  test 
data  from  the  EW  PP  become  available,  post-test  synthesis  and  evaluation  focuses  on 
establishing  demonstrated  values  for  key  system  performance  parameters  and  critical 
technical  performance  parameters.  At  this  point,  test  data  permits  a  more  precise 
statistical  assessment  of  actual  system  performance.  Test  data  from  lOT&E  related 
activities  are  analyzed  to  determine  system  operational  effectiveness  and  suitability. 

This  aids  in  establishing  confidence  levels  for  the  values  and  in  the  development  of 
algorithms  for  fiimre  applications.  Any  developed  simulation  or  model  must,  with  the 
use  of  this  analysis,  be  updated.  Coupling  the  measured  outcomes  of  testing  with  those 
expected  from  simulations  and  models  provides  an  opportunity  to  revalidate  the 
simulations  and  models  and  update  their  parameters  to  improve  the  model's  accuracy. 
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Models  can  be  used  to  evaluate  the  test  data  in  terms  of  military  needs  and  operational 
requirements.  Feedback  to  the  decision  makers  with  information  needed  to  assess  the 
system's  military  worth  and  program  viability  is  key  to  the  process.  The  results  of  T&E 
in  terms  of  assessments  must  be  provided  to  the  PM,  PEO,  MDAs,  and  other  offices  as 
required.  This  is  accomplished  through  DT  and  OT  final  reports  and  briefings.  This 
reporting  enables  the  DoD  EW  T&E  Process  to  provide  analyzed  and  correlated  data  in 
understandable  terms  for  a  valid  program  assessment.  In  addition  to  the  final  reports, 
the  feedback  loop  also  includes  archiving  other  relevant  measured  data  and  outcomes 
along  with  the  test  reports  to  facilitate  the  maintenance  of  a  thorough  history  (audit 
trail)  of  the  program.  This  archiving  process  also  includes  updating  the  SMM  by 
indicating  the  measured  outcomes  resulting  from  testing  the  EW  production  prototype. 

5.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 
Program  Information  to  decide  a  proper  course  of  action.  When  associated  with  an 
acquisition  milestone,  this  decision  is  aimounced  in  an  ADM  which  outlines  the  future 
course  of  action  for  the  program  and  the  SUT.  Additionally,  each  milestone  decision 
contains  the  "exit  criteria"  for  the  next  Phase/Milestone. 

5.5.1  Milestone  III  Decision  Points.  T&E  results  will  be  reported  by  the 
responsible  test  organization.  The  DT&E  report  compares  assessed  values  of  critical 
system  performance  parameters  with  threshold  levels  specified  in  the  Phase  II 

I  Acqui.qition  Program  Baseline  APB  (i.e..  RCM  and  SMM).  It  also  assesses  the 
readiness  of  the  system  for  lOT&E  and  production.  The  OTA  prepares  final  lOT&E 
reports  that  are  timely,  factual,  concise,  complete,  accurate,  and  objective.  The  final 
report  assesses  the  adequacy  of  conducted  OT&E,  and  whether  the  test  and  evaluation 
results  confirm  that  items  or  components  tested  are  operationally  effective  and  suitable 
for  use  in  combat  by  typical  military  users. 

5.5.2  The  Production  Baseline  documents  the  final  performance  thresholds  and 

I  objectives,  as  well  as  costs  and  schedules,  for  the  Production  and  Deployment  Phase  ffl 
of  the  program.  At  Milestone  III,  performance  thresholds  in  the  system  development 
product  specification  should  be  traceable  to  the  performance  thresholds  and  objectives 
documented  in  the  Production  Baseline.  These  values  are  approved  by  the  MDA  at 
Milestone  III. 

5.5.3  Dnciimentation.  Listed  below  is  the  program  documentation  normally 
required  by  the  MDA  to  be  undated  for  Milestone  IIIDoD  Instruction  5000.2  that 
should  be  reviewed  prior  to  this  Phase  and  should  contain  the  latest  updates:  1)  The 
STAR-is  should  be  updated  with  emphasis  on  changed  and/or  updated  critical 
intelligence  parameters  that  may  be  generated  as  new  requirements,  2)  The  ORD,  after 
MS  II,  should  be  modified  only  as  a  result  of  changes  in  the  MNS  or  cost-schedule- 
performance  trade-offs  during  Phase  11.  Key  parameters  from  the  ORD  are  included  in 
the  Production  Baseline  at  MS  III.  The  T&E  managers  are  immediately  notified  of  any 

I  changes  to  ensure  that  system  T&E  can  be  executed  administered  without  delay  and  that 
the  required  T&E  strategy  still  enables  the  SUT  to  render  the  acceptable  degree  of 
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performance  assessment  yaiiditv.  3)  The  IPS  is  updated  to  describe  program-ehanges 
since  MS  II  and  how  the  exit  criteria  in  the  MS  II  ADM  were  satisfied.  It  will  also 
address  risk  reduction  efforts,  any  proposed  preplanned  product  improvement  (P3I) 
recommendations,  program  cost  estimates  and  assessments,  and  the  proposed 
acquisition  strategy  for  the  remainder  of  the  program,-4)-The  TEMP  is  updated,  as 
required,  and  updates  reflect  testing  completed  and  demonstrated  values  attained  to 
date.  The  SMM  and  RCM  should  be  «^updated  and  emphasis  4s-placed  on  completion 
of  DT&E,  its  certification,  and  progress  towards  lOT&E/FOT&E,  At  MS  III,  an 
analysis  of  alternatives  COEA-is  not  required  unless  conditions  have  changed 
sufficiently  so  that  previous  analysis  of  alternatives  cost  effectiveness  determinations 
are  no  longer  valid.  _MS  IE  analyses  of  alternatives  often  only  provide  updated 
estimates  of  life  cycle  costs.  If  a  change  is  of  sufficient  magnitude  to  cause  the  DAB  to 
revisit  its  MS  II  decision,  the  full  MS  II  analysis  of  alternatives -€QEA- is  updated.  The 
T&E  concern  here  is  changes  to  requirements  which  affect  T&E  events  and  actions,  56) 
lOT&E  Report.  This  additional  document  is  required  by  law  to  be  submitted  to  the 
Secretary  of  Defense  and  Congress  for  major  acquisition  programs.  This  is  a  DOT&E 
action. 
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6.  SYSTEMS  T&E  DURING  PHASE  III  -  PRODUCTION^  AND 
FIELDING/DEPLOYMENT.  AND  OPERATIONAL  SUPPORT-ff&D4 


6.0.1  Milestone  Ill-Production  Approval.  The  MS  III  decision  determines 
whether  to  enter  production  and  deploy  the  EW  system(s}  under  development. 

6.0.2  EW  System  T&E  in  Phase 

6.0.2. 1  Pha.<!e  Description.  The  main  objective  of  Phase  Illthe  P&D  phase  is 
to  achieve  an  operational  capability  that  satisfies  mission  needs,  and  ensure  that  an 
efficient  and  stable  production  process  is  established  with  an  adequate  technical  support 
base,  and  to  ensure  ^at  the.  Other  objectives  are  to  conduct  follow-on  operational  and 
production  verification  testing  to  confirm  and  monitor  performance  and  quality  and  to 
verify  the  correction  of  deficiencies.  Deficiencies  encountered  in  Developmental  Test 
and  Evaluation  fDT&F.f  and  Initial  Operational  Test  and  Evaluation  (lOT&Ef  shall  be 
resolved  and  fixes  verified.  During  fielding/deplovment  and  throughout  operational 
support,  the  potential  for  modifications  to  the  fielded/deploved  system  continues.  If? 
during  production,  _a  requirement  for  a  major  modification  is  identified?  that  is  of 
sufficient  cost  and  complexity  that  it  could  itself  qualify  as  an  ACAT I  program,  it  shall 
be  considered  for  management  purposes  as  a  separate  acquisition  effort.  Modifications 
that  do  not  cross  the  ACAT  I  threshold  shall  be  considered  part  of  the  program  being 
modified  an  additional  milestone.  Milestone  IV,  may  be  required.  It  will  be  scheduled  to 
review  and  approve  any  proposed  modifications  to  the  acquisition  strategy,  the  program 
plan,  and  i\PB  (concept,  development  or  production)  and  establish  the  exit' criteria  that 
must  be  met.  If  the  system  is  no  longer  in  production,  the  need  for  an  upgrade  must  be 
established  through  a  new  MNS?^ 

6.0. 2.2  T&E  in  Phase  III  RfeB.  The  objectives  of  T&E  in  Phase 
are  to  assess  the  demonstrate  an  operational  capability  of  the  production  EW  system 
relative  to  that  satisfies-the  mission  need^-asdto  identify  additional  follow-on 
operational  test  and  evaluation  (FOT&E)  that  may  include  follow-on  operational  and 
production  verification  testing  in  accordance  with  the  TEMP,  and  to  conduct 
developmetnal  and  operational  test  and  evaluation  on  non- ACAT  1  level  modifications. 

6.0. 2.2. 1  Developmental  T&E  is  conducted  to  confirm  that 
specifications  are  being  met  on  the  production  version  of  the  system  and  to  evaluate  any 
product  improvement  changes,  such  as  those  resulting  from  ECPs,  changes  in  the 
threat,  efforts  to  reduce  system  life  cycle  costs,  or  efforts  to  improve  system  reliability, 
maintainability,  and  availability  that  are  not  part  of  an  ACAT  I  modifiaction  program. 
This  is  normally  accomplished  through  a  limited  number  of  ground  and  flight  tests.  If 
no  changes  are  made  on  the  initial  production  systems,  there  should  ^let- be  few 
mMQ^additional  DT&E  requirements. 

6.0. 2.2. 2  Operational  T&E  conducted  on  the  production  version  of  the 
system  is  usually  termed  FOT&E.  FOT&E  evaluates  operational  effectiveness  and 
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suitability,  examines  non-ACAT  I  modifications  or  changes,  and  assesses  determines 
whether  the  EW  system  meets  operational  and  logistic  support  needs  in  changing 
operational  environments.  It  is  also  used  1)  to  refine  estimates  of  operational 
effectiveness  and  suitability  made  during  lOT&E,  2)  to  evaluate  changes  made  to 
correct  deficiencies  found  in  prior  T&E,  and  3)  to  identify  potential  additional 
deficiencies.  FOT&E  is  conducted  throughout  the  remainder  of  the  system's  life  and 
further  evaluates  the  system's  operational  effectiveness  and  suitability  to  update 
training,  tactics,  techniques,  and  doctrine  and  to  identify  potential  deficiencies  or  the 
need  for  modifications. 

6.1  STEP  ONE.  Identiflcation  of  T&E  Information  Required  By  Decision 
Makers  from  T&E.  The  main  objective  in  this  phase  is  to  refine  those  EW  system 
concept(s),  operational  effectiveness  and  suitability  performance  parameters,  along  with 
design,  specification,  and  production  parameters  that  must  be  tested  and  evaluated  to 
ensure  that  potential  deficiencies  and  recognized  modifications  can  be  corrected  or 
applied  to  the  system.  The  SMM  and  RCM  should  be  -are  updated  with  any  applicable 
changes  to  thresholds  and  objectives  of  critical  performance  parameters  that  must  be 
accomplished  during  Phase  III. 

6.1.1  Asa  minimum,  T&E  addresses  the  (generic)  questions  as  depicted  in 
Figure  13  and,  more  specifically,  as  listed  below  as  they  might  relate  to  each  EW 
system: 


(1)  For  DT&E,  what  are  the  deployed  system's  capabilities  and  limitations? 
What  are  the 
capabilities 
and  limitations 
of 

modifications 
and  upgrades? 

(2)  For 
OT&E,  does 
the  EW  system 
continue  to  be 
operationally 
suitable  and 

effective  in  operational  use?  Do  proposed  modifications  and  upgrades 
increase  the  suitability  and  effectiveness  of  the  system? 

(3)  Have  performance  objectives  and  thresholds  against  advanced  threats 
been  validated?  Does  the  system  meet  these  performance  objectives  and 
thresholds? 

(4)  Is  the  TEMP  current? 


Phase  Ill-Production,  Fielding/Deployment,  &  Operational  Support 

For  the  selected  system, 

(1)  For  DT&E,  what  are  the  deployed  system's  demonstrated  capabilities  and  limitations? 

What  are  the  capabilities  and  limitations  of  modifications  and  upgrades? 

(2)  For  OT&E,  does  the  system  continue  to  be  operationally  suitable  and  effective  in 
operational  use?  Do  proposed  modifications  and  upgrades  increase  the  suitability  and 
effectiveness  of  the  system? 

(3)  Have  performance  objectives/thresholds  versus  advanced  threats  been  validated?  Does 
the  system  meet  these  performance  objectives/thresholds? 

(4)  Is  the  TEMP  current? 

_ Figure  13.  Phase  III  T&E  Questions 


(5)  Should  a  MS  IV  be  directed  after  MS  III  and,  if  so,  what  are  the  Phase 
IV  T&E  "exit  criteria”? 


6.2  STEP  TWO.  Pre-test  Analysis.  -Afly-Cehanges  in  the  hardware  and  software  at 
this  point  in  a  program  usually  -efien  require  additional  testing  to  verify  and  validate.  As 
before,  the  process  begins  with  pre-test  analysis  to  evaluate  changes  to  the  baseline 
necessary  either  to  correct  noted  deficiencies  or  to  modify  the  system  in  response  to 
threat  changes.  The  data  produced  by  this  analysis  are  used  to  establish  higher 
confidence  expected  values.  The  results  continue  to  be  used  to  support  program 
decisions,  and  the  DSM  is  updated  to  reflect  any  changes  in  demonstrated  performance. 

6.3  STEP  THREE.  Test  Activity  and  Data  Management.  Testing  develops  the  data 
to  assess  an  operational  capability  for  the  production  system  that  satisfies  the  mission 
need  and  identifies  any  additional  required  FOT&E.  Inthe  P&D  Phase;  Phase  III  the 
following  are  considered: 

Historical  Test  Data.  During  Phase  IIIP&B.  the  use  of  historical  data  is  critical. 
Archiving  system  data  in  historical  files  allows  evaluation  of  the  nature  of  an  observed 
deficiency.  If  it  is  observed  only  at  a  single  location,  the  problem  is  probably  not 
systemic  and  the  search  for  solutions  should  first  explore  possible  causes  at  that 
location.  If  the  problem  is  widespread,  a  thorough  review  of  the  nuances  of  the 
deficiency  may  point  more  clearly  to  the  source  of  the  problem  and  may  suggest  or 
even  dictate  the  solution  or  may  prevent  returning  to  a  previous  design  that  was  earlier 
proven  unacceptable. 

Cnmnonent  Measurement  Tests.  Changes  in  the  system's  antennas,  in  antenna 
locations  on  platform(s),  or  in  the  host  platform  signature,  require  additional  static 
measurements.  Measurements  are  also  conducted  during  field  tests  to  obtain  dynamic 
signature  and  antenna  pattern  parameters. 

Integration  Tests.  The  amount  of  SIL  testing  necessary  in  this  phase  depends  on 
the  number,  extent,  and  complexity  of  changes  to  the  system. 

Hardware-In-The-Loop  (HITLl  Tests.  If  modifications  are  made  to  the  system 
prior  to  or  during  production,  additional  HITL  testing  may  be  required.  Changes  to 
countermeasure-response  functions,  in  particular,  usually  require  additional  HITL 
testing  to  verify  that  the  proposed  responses  are  effective. 

Installed  System  Tests.  Installed  system  testing  continues  during  Phase  III  4ie 
P&D  Phase  to  evaluate  the  installed  system's  interfaces  and  interoperability  with  other 
platform  systems.  The  actual  configuration  of  the  system  to  be  deployed  is  tested  in 
installed  system  tests  before  testing  on  open-air  ranges.  This  procedure  helps  in 
identifying  problems,  saves  test  hours,  and  produces  more  usable  test  runs. 
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Field/Ooen  Air  Tests.  Open  air  range  testing  is  often  necessary  to  determine  if 
the  production  configurations  of  the  system  satisfy  user  requirements.  This  testing  may 
include  new  threat  simulators  that  are  more  representative  of  the  actual  threat  and  may 
employ  expanded  test  scenarios  as  more  test  assets  become  available.  These  tests  should 
confirm  whether  system  performance  requirements  established  in  previous  phases  have 
been  achieved. 

.Simnlations/Stimulation  Events.  As  in  the  EMD  Phase,  modeling  and  simulation 
should  be  used  in  Phase  III  P&B  in  lieu  of  actual  testing  where  feasible  to  fill  data 
voids.  When  new  test  scenarios  are  developed  to  respond  to  changes  in  requirements, 
computer  simulations  should  be  used  to  design  new  test  trials  and  estimate  test  results. 

Data  Management.  Data  are  processed  into  meaningful  information  that  can  be 
understood  by  a  person  who  has  some  familiarity  with  the  system  and  the  test 
requirements.  Once  processed,  time  correlated  measurement  values  can  be  used  to 
conduct  a  post-test  assessment  of  the  results. 

6.4  STEP  FOUR.  Post-test  Synthesis  and  Evaluation.  As  higher  quality  test  data 
from  production  systems  become  available,  post-test  synthesis  and  evaluation  focuses 
on  establishing  demonstrated  values  for  key  system  performance  parameters  and  critical 
technical  performance  parameters.  At  this  point,  test  data  permit  a  more  precise 
statistical  analysis  of  achieved  system  performance  and  effectiveness  and  establishes 
confidence  levels  for  performance  parameter  values. 

The  data  provided  as  feedback  are  presented  in  the  form  of  assessments  that  aid  the 
decision  makers  in  reaching  their  decisions  on  the  progress  of  the  SUT.  Full  rate 
production  of  the  system  is  achieved  during  Phase  Illthe  P&D  Phase.  Before  full-rate 
production  is  initiated,  however,  open  ECPs  should  be  evaluated  and  resolved.  ECPs 
may  also  be  generated  due  to  changes  in  threat  and  cost  projections,  or  to  achieve 
reliability  and  maintainability  growth  goals.  Full-Rate  Production  incorporates  the 
changes  generated  by  approved  ECPs  during  the  EMD  Phase.  These  changes  are 
installed  on  new  host  platforms  to  establish  the  initial  deployment  of  the  EW  system. 
Frequently,  a  number  of  the  full-rate  production  systems  are  used  as  test  items  for 
DT&E  and  FOT&E. 

6.5  STEP  FIVE.  Decision  Maker  Weighs  T&E  Information  Against  Other 
Program  Information.  The  output  of  this  step  is  an  informed  decision  on  the  proper 
course  of  action  to  be  taken  at  this  point  predicated  upon  the  information  received  by 
the  decision  maker  from  the  application  of  the  Process  to  date. 
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7.  EW  SYSTEMS  T&E  DURING  PHASE  IV  OPERATIONS  AND  SUPPORT 
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78.  SUMMARY 


The  DoD  T&E  Process  for  Electronic  Warfare  Systems  is  a  disciplined  and 
consistent  process  designed  to  provide  T&E  information  to  the  decision  makers.  This 
information,  when  applied  with  other  program  information,  enables  the  appropriate 
decision  making  elements  to  arrive  at  concrete  decisions  regarding  system  maturation. 

This  Process,  comprised  of  five  steps,  is  iterative  and  assesses  military  worth  in 
terms  of  the  system  contribution  in  meeting  military  needs.  The  T&E  methodology 
provides  for  consistent  testing  of  EW  systems,  provides  for  comprehensive  T&E  of 
performance  objectives  and  thresholds  using  sophisticated  testing  resources,  results  in 
an  accurate  documentation  of  test  result  data,  and  calls  for  timely  analysis  to  be 
forwarded  to  decision  makers  in  the  form  of  DT&E  and  OT&E  assessments.  Consistent 
and  accurate  information  provided  to  decision  makers  is  necessary  to  achieve  timely 
system  development  which  satisfies  user  needs. 

This  T&E  Process  may  be  applied  to  the  acquisition  of  an  EW  system  in  all 
DoD  Components.  It  is  consistent  with  the  DoD  Acquisition  Process  and  appropriately 
defined  within  DoD  5000  series  documentation.  It  is  expressly  intended  to  assist 
acquisition  executives,  program  managers,  program  test  coordinators,  test  organization 
personnel,  and  others  involved  in  the  management  of  T&E  of  new,  or  highly  modified, 
EW  systems. 


443336 


APPENDIX  A 


ABBREVIATIONS  AND  ACRONYMS 


AARM 

Anti-Air  Anti-radiation  Missile 

AASPEM 

Advanced  Air-to-air  System  Performance  Model 

ACAT 

Acquisition  Category 

ADM 

Acquisition  Decision  Memorandum 

ALARM 

Advanced  Low  Altitude  Radar  Model 

APB 

Acquisition  Program  Baseline 

ATD 

Advanced  Technology  Demonstration 

CAIV 

Cost  as  an  Independent  Variable 

CE&D 

CJCS 


COI 

Critical  Operational  Issue 

CTP 

Critical  Technical  Parameter 

C2W 

Command  &  Control  Warfare 

DECM 

Deceptive  Electronic  Countermeasures 

1  DEM/VAL 

— Demonstration  &  Validation  (Phase) 

DIA 

Defense  Intelligence  Agency 

DoD 

Department  of  Defense 

DoDD 

DoD  Directive 

DoDI 

DoD  Instruction 

DSM 

Digital  System  Model 

DSMC 

Defense  Systems  Management  College 

A-l±lr 


DT&E 


EA 

ECCM 

ECM 

ECP 

EMD 

EOA 

EP 

ERP 

ES 

EW 

FOT«&E 

HARM 

HITL 

IOC 

lOT&E 

IPS - 

IR 

IRCM 

JCS 

JC2WC 

JEWC 

J-MASS 


Development  Test  and  Evaluation 
Electronic  Attack 

Electronic  Counter  Countermeasures 

Electronic  Countermeasures 

Engineering  Change  Proposal 

Engineering  and  Manufacturing  Development 

Early  Operational  Assessment 

Electronic  Protection 

Effective  Radiated  Power 

Electronic  Support 

Electronic  Warfare 

Follow-on  Operational  T&E 

High-speed  Anti-Radiation  Missile 

Hardware-in-the-loop 

Initial  Operational  Capability 

Initial  Operational  Test  and  Evaluation 
Integrated  Program  Summary 

Infra-red 

Infra-red  Countermeasures 
Joint  Chiefs  of  Staff 

Joint  Command  and  Control  Warfare  Center  (formerly  JEWC) 
Joint  Electronic  Warfare  Center 
Joint  Modeling  and  Simulation  System 


A-244 


MDA 

Milestone  Decision  Authority 

MNS 

Mission  Need  Statement 

MOE 

Measurement  of  Effectiveness 

MOP 

Measure  of  Performance,  Memorandum  of  Policy 

MS 

Milestone 

MTI 

Moving  Target  Indicator 

OPSEC 

Operational  Security 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

OTA 

Operational  Test  Agency 

OT&E 

Operational  T&E 

PDA 

Program  Decision  Authority 

PEO 

Program  Executive  Officer 

PM 

Program  Manager 

PP 

Production  Prototype 

PSYOP 

Psychological  Operations 

P3I 

Preplanned  Product  Improvement 

RAM 

Reliability,  Availability,  and  Maintainability 

RCM 

Requirements  Correlation  Matrix 

RCS 

Radar  Cross  Section 

RDA 

Research,  Development,  and  Acquisition 

SAM 

Surface-to-Air  Missile 
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SEAD 

Suppression  of  Enemy  Air  Defenses 

SIL 

System  Integration  Laboratory 

SMM 

System  Maturity  Matrix 

STAR 

System  Threat  Assessment-Repeft 

SUT 

System  Under  Test 

TEMP 

T&E  Master  Plan 

TST 

Tactical  SIGINT  Technology 

USD(A&T) 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 

WARM 

Wartime  Reserve  Modes 
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APPENDIX  B 


DEFINITIONS 


ACQUISITION  CATEGORY  (ACAT)  Categories  established  for  acquisition 
programs  to  facilitate  decentralized  decision  making  and  execution  and  compliance  with 
statutorily  imposed  requirements.  The  categories  determine  the  level  of  review, 

I  decision  authority,  and  applicable  procedures.  (DoD  5000.2-RDoDI  5000.2) 

a.  ACAT  I.  Has  been  designated  by  the  Under  Secretary  of  Defense 
(Acquisition  and  Technology)  as  an  ACAT  I  program  or  has  an  eventual  RDT&E 
expenditure  of  more  than  $200  million  and  a  procurement  expenditure  of  more  than  $1 
billion  in  1980  constant  dollars. 

b.  ACAT  II.  Has  been  designated  by  the  DoD  head  component  as  an  ACAT  11 
I  program.  Has  an  eventual  RDT&EK&&E  expenditure  of  more  than  $75  million  and  a 

procurement  expenditure  of  $300  Million  in  1980  constant  dollars. 

c.  ACAT  III.  Programs  not  meeting  the  criteria  for  ACAT  I  or  II  that  have 
been  designated  Category  11  by  the  DoD  Component  Acquisition  Executive. 

d.  ACAT  IV.  Other  acquisition  programs  for  which  the  milestone  decision 
authority  should  be  delegated  to  a  level  below  that  required  for  ACAT  III. 

ACQUISITION  PROCESS  The  basis  for  comprehensive  management  and  the 
progressive  decision  making  associated  with  program  maturation.  The  acquisition 
process  consists  of  five  major  milestone  decision  points  and  five  phases  (Concept 
I  Explorationand  Definitiea,  Demonstration  and  Validation,  Engineering  and 
Manufacturing  Development,  Production  and  Deployment,  and  Operations  and 
1  Support).,  (DoD  5000.2-RDoDI  5000.2) 

ACQUISITION  DECISION  MEMORANDUM  (ADM)  A  memorandum  signed  by 
the  milestone  decision  authority  that  documents  decisions  made  and  the  exit  criteria 
established  as  the  result  of  a  milestone  decision  review  or  in-process  review.,  (DoD 
5000.2-RDoDI  5000.3) 

ACQUISITION  PLAN  A  formal  written  document  reflecting  the  specific  actions 
necessary  to  execute  the  approach  established  in  the  approved  acquisition  strategy  and 
guiding  contractual  implementation.  (FAR  Subpara.  7.1) 
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ACQUISITION  PROGRAM  A  directed,  funded  effort  that  is  designed  to  provide  a 
new  or  improved  materiel  capability  in  response  to  a  validated  need.  (DoDD  5000. 1) 

ACQUISITION  PROGRAM  BASELINE  (APB)  A  document  required  for  all 
acquisition  categories  that  embodies  the  cost,  schedule,  and  performance  objectives  for 
the  program.  (BoD  5000.2-RDoDI  5000:^) 

ACQUISITION  STRATEGY  The  master  plan  for  program  execution  from  program 
initiation  through  post-production  support.  It  is  to  be  developed  in  sufficient  detail  to 
establish  the  managerial  approach  that  will  be  used  to  direct  and  control  all  elements  of 
the  acquisition  to  achieve  program  objectives.  (BoD  5000.2-RDoDI  5000.2) 

ANTI-RADIATION  MISSILE  A  missile  which  homes  passively  on  a  radiation 
source.  (CJCS  MOP  6) 

AVAILABILITY  A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and 
committable  state  at  the  start  of  a  mission  when  the  mission  is  called  for  at  an  unknown 
(random)  time.  fPoD  5000.2-RDoDI  5000.2) 

AVIONICS  Electrical  and  electronic  systems  and  devices  used  in  aviation,  missilery, 
and  astronautics.  (Commonly  used  military  jargon  that  is  a  contraction  of  nation 
electronics)  (EW  T&E  Task  Force) 

BRASSBOARD  CONFIGURATION  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  and  operational  data.  It  normally 
is  a  model  sufficiently  hardened  for  use  outside  of  laboratory  environments  to 
demonstrate  the  technical  and  operational  principles  of  immediate  interest.  It  may 
resemble  the  end  item,  but  is  not  intended  for  use  as  the  end  item.  (EW  T&E  Task 
Force) 

BREADBOARD  CONFIGURATION  An  experimental  device  (or  group  of  devices) 
used  to  determine  feasibility  and  to  develop  technical  data.  It  normally  is  configured 
only  for  laboratory  use  to  demonstrate  the  technical  principles  of  immediate  interest.  It 
may  not  resemble  the  end  item  and  is  not  intended  for  use  as  the  projected  end  item. 
(EW  T&E  Task  Force) 

CLOSED  LOOP  A  term  used  frequently  in  testing  electronic  systems  to  characterize  a 
situation  in  which  both  the  SUT  and  a  stimulating  system  (usually  a  threat  system)  can 
mutually  respond  and  interact,  and  the  responses  and/or  behavior  of  both  are  measured 
or  recorded.  (EW  T&E  Task  Force) 

COMMAND  AND  CONTROL  (C^)  The  exercise  of  authority  and  direction  by  a 
properly  designated  commander  over  assigned  or  attaehed  forces  in  the  accomplishment 
of  the  mission.  C^  functions  are  performed  through  an  arrangement  of  personnel. 


equipment,  communications,  computers,  facilities,  and  procedures  employed  by  a 
commander  in  planning,  directing,  coordinating,  and  controlling  forces  and  operations 
in  accomplishment  of  the  mission.  (Joint  Pub  1-02) 

COMMONALITY  A  quality  which  applies  to  materiel  or  systems:  a.  possessing  like 
and  interchangeable  characteristics  enabling  each  to  be  utilized,  or  operated  and 
maintained,  by  personnel  trained  on  the  others  without  additional  specialized  training; 
b.  having  interchangeable  parts  and/or  components;  or  c.  applying  to  consumable  items 
interchangeably  equivalent  without  adjustment.  (Joint  Pub  1-02) 

COMPATIBILITY  The  capability  of  two  or  more  items  or  components  of  equipment 
or  material  to  exist  or  function  in  the  same  system  or  environment  without  mutual 
interference.  (Joint  Pub  1-02) 

CONCEPT  A  notion  or  statement  of  an  idea,  expressing  how  something  might  be 
done  or  accomplished,  that  may  lead  to  an  accepted  procedure.  (Joint  Pub  1-02) 

CONCEPT  BASELINE  The  baseline  approved  at  Milestone  I  that  contains  broad 
objectives  and  thresholds  for  key  cost,  schedule,  and  performance  parameters.  It  is 
submitted  by  the  designated  component  official  through  the  MDA  as  a  stand-alone  part 
of  the  MS  I  documentation.  (BoD  5000.2-RDoDI  5000.2) 

ANALYSIS  OF  ALTERNATIVES  COST  AND  OPERATIONAL 
EFFECTIVENESS  ANALYSIS  (COEA)  An  analysis  of  the  estimated  costs  and 
operational  effectiveness  of  alternative  materiel  systems  to  meet  a  mission  need  and  the 
associated  program  for  acquiring  each  alternative.  (BoD  5000.2-RDoDI  5000.2) 

COST  AS  AN  INDEPENDENT  VARIABLE  (CAIV)  The  acquisition  strategy  shall 
address  methodologies  to  acquire  and  operate  affordable  BoB  systems  by  setting 
aggressive,  achievable  cost  objectives  and  managing  achievement  of  these  objectives. 
Cost  obieo-tives  shall  he  set  to  balance  mission  needs  with  projected  out-year  resources, 
takiriP  into  account  anticipated  process  improvements  in  both  BoB  and  defense 
industries.  (BoB  5000. 2-R) 

COST  EFFECTIVENESS  A  measure  of  the  operational  capability  added  by  a  system 
as  a  function  of  its  life-cycle  cost.  (B-(oB  5000.2-RBoBI  50(X).'2) 

CRITICAL  DESIGN  REVIEW  (CDR)  A  review  conducted  to  determine  that  the 
detailed  design  satisfies  the  performance  and  engineering  requirements  of  the 
development  specification;  to  establish  the  detailed  design  compatibility  among  the  item 
and  other  items  of  equipment,  facilities,  computer  programs,  and  personnel;  to  assess 
producibility  and  risk  areas;  and  to  review  the  preliminary  product  specifications. 
Conducted  during  Phase  I,  Phase  I  BEM/VAL  (for  prototypes)  and  Phase  II,  EMB. 
(BoB  5000.2-RBoBI  5000:2) 


CRITICAL  OPERATIONAL  ISSUE  (COI)  A  key  operational  effectiveness  or 
operational  suitability  issue  that  must  be  examined  in  operational  test  and  evaluation  to 
determine  the  system's  capability  to  perform  its  mission.  A  critical  operational  issue  is 
normally  phrased  as  a  question  to  be  answered  in  evaluating  a  system's  operational 
effectiveness  and/or  operational  suitability.  (BoD  5000.2-RDoDI  5000.2) 

CRITICAL  SYSTEM  CHARACTERISTICS  Design  features  that  determine  how 
well  the  proposed  concept  or  system  will  function  in  its  intended  operational 
environment.  They  include  survivability;  transportability;  energy  efficiency;  and 
interoperability,  standardization,  and  compatibility  with  other  forces  and  systems 
including  support  infrastructure.  fPoD  5000.2-RDoDI  5000.2) 

CRITICAL  TECHNICAL  PARAMETERS  Design-specific  system  performance 
and/or  technical  parameters  characterizing  system  capabilities  required  to  achieve  key 
performance  threshold  values.  (DoD  5000.2-RDoDI  5000.2) 

DESIGN  The  process  of  converting  specifications  into  a  visual  or  mathematical 
representation  of  the  system.  (EW  T&E  Task  Force) 

DETECTION  The  perception  of  an  object  of  possible  military  interest  but 
unconfirmed  by  recognition.  (Joint  Pub  1-02) 

DEVELOPMENT  BASELINE  Baseline  containing  more  detailed  and  refined 
objectives  and  thresholds  for  key  cost,  schedules  and  performance  parameters  then  the 
concept  baseline.  It  is  submitted  as  a  stand-alone  part  of  the  Milestone  n 
documentation.  (DoD  5000.2-RDoDI  5000.2) 

DEVELOPMENT  TEST  AND  EVALUATION  (DT&E)  Activity  that  is  conducted  to 
demonstrate  that  the  engineering  design  and  development  process  is  complete.  It  is  used 
to  reduce  risk,  validate  and  qualify  system  design  and  to  ensure  that  the  end-product  is 
ready  for  operational  test  and  evaluation.  The  DT&E  is  the  responsibility  of  the 
materiel  developer.  (DSMC) 

DIRECTED  ENERGY  An  umbrella  term  covering  technologies  that  relate  to  the 
production  of  a  beam  of  concentrated  electromagnetic  energy  or  atomic  or  subatomic 
particles.  Also  called  DE.  (CJCS  MOP  6) 

DIRECTED-ENERGY  WEAPON  A  system  using  directed-energy  primarily  as  a 
direct  means  to  damage  or  destroy  enemy  equipment,  facilities,  and  personnel.  (CJCS 
MOP  6) 

DoD  COMPONENT  ACQUISITION  EXECUTIVE  A  single  official  within  a  DoD 
Component  who  is  responsible  for  all  acquisition  functions  within  that  Component. 
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This  includes  Service  Acquisition  Executives  for  the  Military  Departments  and 
Acquisition  Executives  in  other  DoD  Components  who  have  acquisition  management 
I  responsibilities.  tPoD  5000.2-RDoDI  5000.2) 

EARLY  OPERATIONAL  ASSESSMENT  An  operational  assessment  conducted 
I  prior  to,  or  in  support  of,  Milestone  n._(DoD  5000.2) 

ELECTROMAGNETIC  COMPATIBILITY  (EMC)  The  ability  of  systems 
equipment,  and  devices  that  utilize  the  electromagnetic  spectrum  to  operate  in  their 
intended  operational  environments  without  suffering  unacceptable  degradation  or 
causing  unintentional  degradation  because  of  electromagnetic  radiation  or  response.  It 
involves  the  application  of  sound  electromagnetic  spectrum  management;  system, 
equipment,  and  device  design  configuration  that  ensures  interference-free  operation; 
and  clear  concepts  and  doctrines  that  maximize  operational  effectiveness.  (CJCS  MOP 
6) 

ELECTROMAGNETIC  DECEPTION  The  deliberate  radiation,  re-radiation, 
alteration,  suppression,  absorption,  denial,  enhancement,  or  reflection  of 
electromagnetic  energy  in  a  manner  intended  to  convey  misleading  information  to  an 
enemy  or  to  enemy  electromagnetic-dependent  weapons  thereby  degrading  or 
neutralizing  the  enemy's  combat  capability.  (CJCS  MOP  6) 

ELECTROMAGNETIC  ENVIRONMENT  (EME)  The  resulting  product  of  the 
power  and  time  distribution,  in  various  frequency  ranges,  of  the  radiated  or  conducted 
electromagnetic  emission  levels  that  may  be  encountered  by  a  military  force,  system,  or 
platform  when  performing  its  assigned  mission  in  its  intended  operational  environment. 
(Joint  Pub  1-02) 

ELECTROMAGNETIC  HARDENING  Action  taken  to  protect  personnel,  facilities, 
and/or  equipment  by  filtering,  attenuating,  grounding,  bonding,  and/or  shielding 
against  undesirable  effects  of  electromagnetic  energy.  (CJCS  MOP  6) 

ELECTROMAGNETIC  INTERFERENCE  (EMI)  Any  electromagnetic  disturbance 
that  interrupts,  obstructs,  or  otherwise  degrades  or  limits  the  effective  performance  of 
electronics/electrical  equipment.  It  can  be  induced  intentionally,  as  in  some  forms  of 
electronic  warfare,  or  unintentionally,  as  a  result  of  spurious  emissions  and  responses, 
intermodulation  products,  and  the  like.  (CJCS  MOP  6) 

ELECTROMAGNETIC  JAMMING  The  deliberate  radiation,  re-radiation,  or 
reflection  of  electromagnetic  energy  for  the  purpose  of  preventing  or  reducing  an 
enemy's  effective  use  of  the  electromagnetic  spectrum,  and  with  the  intent  of  degrading 
or  neutralizing  the  enemy's  combat  capability.  (CJCS  MOP  6) 


ELECTRONIC  COUNTER-COUNTERMEASURES  (ECCM)  The  division  of 
electronic  warfare  involving  actions  taken  to  ensure  friendly  effective  use  of  the 
electromagnetic  spectrum  despite  the  enemy's  use  of  electronic  warfare.  (Joint  Pub  1- 
02) 

ELECTRONIC  COUNTERMEASURES  (ECM)  The  division  of  electronic  warfare 
involving  action  taken  to  prevent  or  reduce  the  enemy's  effective  use  of  the 
electromagnetic  spectrum.  It  includes  electronic  jamming  and  electronic  deception. 
(Joint  Pub  1-02) 

ELECTRONIC  MASKING  The  controlled  radiation  of  electromagnetic  energy  on 
friendly  frequencies  in  a  manner  to  protect  the  emissions  of  friendly  communications 
and  electronic  systems  against  enemy  ESM/SIGINT  (electronic  warfare  support 
measures/signals  intelligence),  without  significantly  degrading  the  operation  of  friendly 
systems.  (CJCS  MOP  6) 

ELECTRONIC  WARFARE  EXPENDABLES  Nonrecoverable  EW  devices  such  as 
chaff,  flares,  unmanned  vehicles,  decoys,  and  unattended  jammers.  (CJCS  MOP  6) 

ELECTRONICS  INTELLIGENCE  (FLINT)  Technical  and  intelligence  information 
derived  from  foreign  non-communications  electromagnetic  radiations  emanating  from 
other  than  nuclear  detonations  or  radioactive  sources.  (CJCS  MOP  6) 

EMISSION  CONTROL  (EMCON)  The  selective  and  controlled  use  of 
electromagnetic,  acoustic,  or  other  emitters  to  optimize  command  and  control 
capabilities  while  minimizing,  for  operations  security  (OPSEC),  detection  by  enemy 
sensors;  to  minimize  mutual  interference  among  friendly  systems;  and/or  to  execute  a 
military  deception  plan.  (CJCS  MOP  6) 

ENGINEERING  CHANGE  PROPOSAL  (ECP)  A  formal  document  used  to  make 
engineering  changes  in  an  existing  contract.  (DSMC) 

ENGINEERING  PROTOTYPE  (EP)  A  development  model  of  a  unit  that  is  close  to 
production.  The  term  may  apply  to  circuitry,  a  device  (black  box)  or  a  system,  and 
may  be  in  a  breadboard  (technical)  configuration.  (EW  T&E  Task  Force) 

EVALUATION  The  technical  and/or  operational  study  and  investigations  by  a 
developing  and/or  operational  agency  to  determine  the  technical  and/or  operational 
suitability  and  effectiveness  of  material,  equipment,  or  a  system  for  use  in  the  military 
Services.  (Joint  Pub  1-02) 

EVALUATION  CRITERIA  Standards  by  which  accomplishments  of  required 
technical  and  operational  effectiveness  and/or  suitability  characteristics  or  resolution  of 
operational  issues  may  be  assessed.  (DoD  5000. 2-R) 


EXIT  CRITERIA  Program  specific  accomplishments  that  must  be  satisfactorily 
demonstrated  before  an  effort  or  program  can  proceed  further  in  the  current  acquisition 
phase  or  transition  to  the  next  acquisition  phase.  Exit  criteria  may  include  such  factors 
as  critical  test  issues,  the  attainment  of  projected  growth  curves  and  baseline 
parameters,  and  the  decision  to  proceed  further.  Exit  criteria  are  specific  to  each 
I  acquisition  phase.  DoD  5000 ■2-R) 

FOLLOW  ON  OPERATIONAL  TEST  AND  EVALUATION  Test  and  evaluation 
that  is  necessary  during  and  after  the  production  period  to  refine  the  estimates  made 
during  operational  test  and  evaluation,  to  evaluate  changes,  and  to  reevaluate  the  system 
to  ensure  it  continues  to  meet  operational  needs  and  retains  its  effectiveness  in  a  new 
1  environment  or  against  a  new  threat.  (DoD  5000. 2-R) 

FULL  RATE  PRODUCTION  Production  of  economic  quantities  following 
stabilization  of  the  system  design  and  prove-out  of  the  production  process.  (DoD 
I  5000.2:R) 

IDENTIFICATION  The  process  of  determining  the  friendly  or  hostile  character  of  an 
unknown  detected  contact.  (Joint  Pub  1-02) 

IMITATIVE  ELECTROMAGNETIC  DECEPTION  The  introduction  of 
electromagnetic  energy  into  enemy  systems  that  imitates  enemy  emissions.  (CJCS 
MOP  6) 

INITIAL  OPERATIONAL  CAPABILITY  The  first  attainment  of  the  capability  to 
employ  effectively  a  weapon,  item  of  equipment,  or  system  of  approved  specific 
characteristics,  and  which  is  manned  or  operated  by  a  trained,  equipped,  and  supported 
I  military  unit  or  force.  (DoD  5000.2-RDoDI-50QQt2) 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (lOT&E)  All  operational 
test  and  evaluation  conducted  on  production  or  production  representative  articles,  to 
support  the  decision  to  proceed  beyond  low-rate  initial  production.  It  is  conducted  to 
provide  a  valid  estimate  of  expected  system  operational  effectiveness  and  operational 
I  suitability.  (DoD  5000.2-RDoDI  5000.2) 

INSTRUMENTATION  The  installation  and  use  of  electronic,  gyroscopic,  and  other 
instruments  for  the  purpose  of  detecting,  measuring,  recording,  telemetering, 
processing,  or  analyzing  different  values  or  quantities  as  encountered  in  the  flight  of  an 
aircraft,  missile,  or  spacecraft.  Instrumentation  applies  to  flight-borne,  sea-borne,  and 
ground-based  equipment.  (AFM  11-1) 

INTEGRi\TED  PROGRAIM  SUMMARY  (IPS)  A  DoD  component  document 
prepared  and  submitted  to  the  milestone  decision  authority  in  support  of  Milestone  I IV 
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reviews.  It  succinctly  highlights  tho  status  of  the  program  and  its  readiness  to  proceed 
into  the  next  phase  of  the  acquisition  process.  (DoDI  5000.2) 

INTEROPERABILITY  The  ability  of  systems,  units,  or  forces  to  provide  services  to 
or  accept  services  from  other  systems,  units,  or  forces  and  to  use  the  services  so 
exchanged  to  operate  effectively  together.  (BoD  5000.2-RDoDI  5000.2) 

JOINT  REQUIREMENTS  OVERSIGHT  COUNCIL  (JROC)  The  JROC  is 
responsible  to  the  Chairman  of  the  Joint  Chiefs  of  Staff  for  assessing  military 
requirements  in  support  of  the  defense  acquisition  process.  The  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff  chairs  the  Council  and  decides  all  matters  before  the  Council.  The 
permanent  members  include  the  Vice  Chiefs  of  the  Army  and  Air  Force,  the  Vice  Chief 
of  Naval  Operations,  and  the  Assistant  Commandant  of  the  Marine  Corps.  The  Council 
directly  support  the  Defense  Acquisition  Board  through  the  review,  validation,  and 
approval  of  military  requirements  at  the  start  of  the  acquisition  process,  prior  to  each 
milestone  review,  or  as  requested  by  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology.  (BoD  5000.2-RDoDI  5000.2) 

KEY  PARAMETERS  Those  parameters  that,  if  the  thresholds  are  not  met,  the 
milestone  decision  authority  would  require  a  reevaluation  of  alternative  concepts  or 
design  approaches.  They  are  derived  from  the  ORD  and  included  as  thresholds  in 
baseline  documentation.  (DoD  5000.2-RDoDI  5000.2) 

LIFE-CYCLE  COST  The  total  cost  to  the  Government  of  acquisition  and  ownership 
of  that  system  over  its  useful  life.  It  includes  the  cost  of  development,  acquisition, 
support  and,  where  applicable,  disposal.  ADoD  5000.2-RDoDI  5000.-2) 

LIVE  FIRE  TEST  AND  EVALUATION  REPORT  A  report  to  be  submitted  by  the 
DOT&E  and  approved  bv  the  Secretary  of  Defense  (or  as  delegated  to  the  Under 
SocrQtnry  of  Dofonse  for  the  USD(A&T)  for  applicable  (covered  under  the  live  fire  test 
law)  ACAT  I  and  II  programs  or  tho  Diroctor,  Dofenoe  Research  &  Engineering,  for 
applicable  ACAT  II  programs)  to  the  Committees  on  Armed  Services  and  on 
Appropriations  of  the  Senate  and  the  House  of  Representatives  prior  to  a  decision  to 
proceed  beyond  low-rate  initial  production  (LRIP).  It  is  also  required  for  a  covered 
product  improvement  program  of  any  acquisition  category  which  is  likely  to 
significantly  affect  the  survivability  of  a  covered  major  system  or  the  lethality  of  a 
major  munition  or  missile  produced  under  an  ACAT  I  or  11  program.  (DoD  5000.2- 
RDoDI  5000.-3) 

LIVE  SIMULATION  A  component  of  simulation  that  involves  operations  with  real 
force  or  personnel  and  real  equipment  or  test  items  in  the  air,  on  the  ground,  on  or 
below  the  sea,  or  in  a  test  facility.  (Defense  Science  Board) 
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LOGISTICS  SUPPORT  ABILITY  The  degree  to  which  planned  logistics  support 
(including  test,  measurement,  and  diagnostic  equipment;  spares  and  repair  parts; 
technical  data;  support  facilities;  transportation  requirements;  training;  manpower;  and 
software  support)  allow  meeting  system  availability  and  wartime  usage  requirements. 
(BoD  5OQO.2-RD0DI  5000;3) 

LOW  RATE  INITIAL  PRODUCTION  (LRIP)  The  production  of  a  system  in 
limited  quantity  to  provide  articles  for  OT&E,  to  establish  an  initial  production  base, 
and  to  permit  an  orderly  increase  in  the  production  rate  sufficient  to  lead  to  full-rate 
production  upon  successful  completion  of  operational  testing.  (DoD  50(K).2-R) 

MAINTAINABILITY  The  ability  of  an  item  to  be  retained  in  or  restored  to  a 
specified  condition  when  maintenance  is  performed  by  personnel  having  specified  skill 
levels,  using  prescribed  procedures  and  resources,  at  each  prescribed  level  of 
maintenance  and  repair.,  (DoD  5000.2-RDoDI  5000:3) 

MAJOR  MODIFICATION  A  modification  that  in  and  of  itself  meets  the  criteria  of 
acquisition  category  (ACAT)  I  or  II  or  is  designated  as  such  by  the  milestone  reviewing 
authority.  A  major  modification  is  necessitated  by:  1)  a  change  in  the  threat  or  Defense 
Planning  Guidance,  2)  a  deficiency  identified  during  follow-on  operational  testing,  or  3) 
operational  training  and  support,  or  an  opportunity  to  reduce  the  cost  of  system 
ownership.  Within  the  context  of  this  EW  T«feE  process,  the  need  for  a  major 
modification  would  dictate  that  the  system  reenter  the  acquisition  process  at  Phase  0  and 
restart  with  the  definition  of  requirements  to  begin  a  new  concepmal  design.  (DoD 
5000.2-RDoDI  5000.2) 

MANIPULATIVE  ELECTROMAGNETIC  DECEPTION  -Actions  to  eliminate 
revealing,  or  convey  misleading,  electromagnetic  telltale  indicators  that  may  be  used  by 
hostile  forces.  (CJCS  MOP  6) 

MEASURES  OF  EFFECTIVENESS  (MOEs)  Tools  that  assist  in  discriminating 
among  a  number  of  alternatives,  normally  to  answer  a  Critical  Operational  Issue  (COI). 
They  show  how  the  alternatives  compare  in  meeting  functional  objectives  and  mission 
needs.  They  are  "predictions"  of  how  a  system  will  perform  when  fielded,  and  are 
usually  expressed  as  a  probability  or  likelihood,  and  usually  inferred.  (DoD  5000.2- 
M) 

MEASURES  OF  PERFORMANCE  (MOPs)  MOPs  are  quantitative  and  qualitative 
measures  of  a  system's  performance  or  system  characteristic.  MOPs  indicate  the 
degree  to  which  a  system  performs  the  task  or  meets  a  requirement  under  specified 
conditions.  MOPs  should  address  system  capabilities  and  characteristics  and  should 
relate  to  the  MOE(s)  such  that  the  effect  of  a  change  in  the  MOP  can  be  related  to  a 
change  in  the  MOE.  (AR  73-1) 
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MEASUREMENT  AND  SIGNATURE  INTELLIGENCE  (MASBVT)  Scientific  and 
technical  intelligence  information  obtained  by  quantitative  and  qualitative  analysis  of 
data  (metric,  angle,  spatial,  wavelength,  time  dependence,  modulation,  plasma,  and 
hydromagnetic)  derived  from  specific  technical  sensors  for  the  purpose  of  identifying 
any  distinctive  features  associated  with  the  source,  emitter,  or  sender  and  to  facilitate 
subsequent  identification  and/or  measurement  of  the  same.  (CJCS  MOP  6) 

MILESTONE  DECISION  AUTHORITY  The  individual  designated  in  accordance 
with  criteria  established  by  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  to  approve  entry  of  an  acquisition  program  into  the  next  phase.  (DoDD 

5000.1) 

MILESTONES  Major  decision  points  that  separate  the  phases  of  an  acquisition 
program.  (DoDD  5000.1) 

MILITARY  DECEPTION  Actions  executed  to  mislead  foreign  decision  makers, 
causing  them  to  derive  and  accept  desired  appreciations  of  military  capabilities, 
intentions,  operations,  or  other  activities  that  evoke  foreign  actions  that  contribute  to 
the  originator's  objective.  (CJCS  MOP  30) 

MINIMUM  ACCEPTABLE  OPERATIONAL  REQUIREMENT  The  value  for  a 
performance  or  technical  parameter  that  is  required  to  provide  a  system  capability  that 
will  satisfy  the  validated  mission  need.  Also  known  as  the  performance  threshold. 
tPoD  5000.2-RDoDI  5000.2) 

MISSION  NEEDS  STATEMENT  (MNS)  A  statement,  expressed  in  broad 
operational  terms,  of  operational  capability  required  to  perform  an  assigned  mission  or 
to  correct  a  deficiency  in  existing  capability  to  perform  the  mission.  (DoD  5000.2- 
RDoDI  50007^) 

MISSION  RELIABILITY  The  probability  that  the  system  will  perform  mission 
essential  functions  for  a  period  of  time  under  the  conditions  stated  in  the  mission 
profile.  (DoD  5000.2-RDoDI  5000.-2) 

MODEL  A  representation  of  an  acmal  or  conceptual  system  that  involves 
mathematics,  logical  expressions,  or  computer  simulations  that  can  be  used  to  predict 
how  the  system  might  perform  or  survive  under  various  conditions  or  in  a  range  of 
hostile  environments.  (DoD  5000.2-RDoDI  5000.2) 

MODIFICATION  A  change  to  a  system  (whether  for  safety,  to  correct  a  deficiency, 
or  to  improve  program  performance)  that  is  still  being  produced.  (DoD  5Q00.2-RBeDI 

5000.2) 
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NONDESTRUCTIVE  ELECTRONIC  WARFARE  -Those  EW  actions,  not 
including  employment  of  WARM,  that  deny,  disrupt,  or  deceive  rather  than  damage  or 
destroy.  (CJCS  MOP  6) 

NON-DEVELOPMENT AL  ITEM  1.  Any  item  of  supply  that  is  available  in  the 
commercial  marketplace;  2.  Any  previously  developed  item  of  supply  that  is  in  use  by  a 
department  or  agency  of  the  United  States,  a  State  or  local  government,  or  a  foreign 
government  with  which  the  United  States  has  a  mutual  defense  cooperation  agreement; 
3.  Any  item  of  supply  described  in  definition  1  or  2.,  above,  that  requires  only  minor 
modification  in  order  to  meet  the  requirements  of  the  procuring  agency;  or  4.  Any  item 
of  supply  that  is  currently  being  produced  that  does  not  meet  the  requirements  of 
definition  1.,  2.,  or  3.  above,  solely  because  of  the  item  is  not  yet  in  use  or  is  not  yet 
available  in  the  commercial  marketplace.  (BoD  5000.2-RDoDI  5000.2) 

OBJECTIVE  Value  beyond  the  threshold  that  could  potentially  have  measurable, 
beneficial  impact  on  capability  or  operations  and  support  above  that  provided  by  the 
threshold  value.  (DoD  5000.2-RDoDI  5000.2) 

OPEN-AIR  TEST  Testing  performed  in  an  outdoor  operating  environment,  i.e.,  on  an 
open-air  test  range  or  on  an  airborne  platform.  (EW  T&E  Task  Force) 

OPEN  LOOP  A  test  scenario  in  which  only  one  system  is  allowed  to  interact  to 
another's  actions.  For  example,  in  the  test  of  jammer  against  a  SAM  system,  emissions 
from  the  SAM  are  received  by  the  jammer,  which  begins  jamming.  The  simulated 
SAM  system,  however,  is  not  allowed  to  receive  or  react  to  this  jamming.  (EW  T&E 
Task  Force) 

OPERATIONAL  ASSESSMENT  An  evaluation  of  operational  effectiveness  and 
operational  suitability  made  by  an  independent  operational  test  activity,  with  user 
support  as  required,  on  other  than  production  systems.  The  focus  of  an  operational 
assessment  is  on  significant  trends  noted  in  development  efforts,  programmatic  voids, 
areas  of  risk,  adequacy  of  requirements,  and  the  ability  of  the  program  to  support 
adequate  operational  testing.  Operational  assessments  may  be  made  at  any  time  using 
technology  demonstrators,  prototypes,  mockups,  engineering  development  models,  or 
simulations  but  will  not  substitute  for  the  independent  operational  test  and  evaluation 
necessary  to  support  full  production  decisions.  (BoD  5000. 2-RDoDI  5000.2) 
OPERATIONAL  CONCEPT  The  way  in  which  forces  and  equipment  are  arranged 
and  employed  in  battle.  This  includes  both  doctrine  and  tactics  concerning  how-a 
system  would  be  used  to  accomplish  national  objectives.  (BoBB  5000.2  M) 

OPERATIONAL  EFFECTIVENESS  The  overall  degree  of  mission  accomplishment 
of  a  system  when  used  by  representative  personnel  in  the  environment  planned  or 
expected  (e.g.,  natural,  electronic,  threat,  etc.)  for  operational  employment  of  the 
system  considering  organization,  doctrine,  tactics,  survivability,  vulnerability,  and 
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threat  (including  countermeasures,  initial  nuclear  weapons  effects,  nuclear,  biological, 
and  chemical  contamination  (NBCC)  threats).  (DoD  5000.2-RDoDI  5000.2) 

OPERATIONAL  REQUIREMENT  An  established  need  justifying  the  timely 
allocation  of  resources  to  achieve  a  capability  to  accomplish  approved  military 
objectives.  (Joint  Pub  1-02) 

OPERATIONAL  REQUIREMENTS  DOCUMENT  (ORD)  A  formatted  statement 
containing  performance  (operational  effectiveness  and  suitability)  and  related 
operational  parameters  for  the  proposed  concept  or  system.  It  is  used  to  develop 
requirements  for  contract  specifications  during  each  acquisition  phase.  The  ORD  is 
initially  prepared  by  the  user  or  user's  representative  during  ^hase  0,  Concept 
Exploration  and  Definition,  for  the  preferred  concept(s)  to  be  proposed  at  Milestone  I. 
(DoD  5Q00.2-RDOD  5000.2  M) 

OPERATIONAL  SUITABILITY  The  degree  to  which  a  system  can  be  placed 
satisfactorily  in  field  use,  considering  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors, 
manpower  supportability,  logistics  supportability,  natural  environmental  effects  and 
impacts,  documentation,  and  training  requirements.  (EMoD  5000.2-RDoDI  5000.2) 

OPERATIONAL  TEST  AGENCY  (OTA)  The  command  or  agency  designated  by  the 
program  management  directive,  or  other  appropriate  directive,  as  responsible  for 
managing  and  conducting  the  independent  OT&E  of  a  system.  (DoD  5000.2-RDeDI 
5000.-^) 

OPERATIONAL  TEST  AND  EVALUATION  (OT&E)  Test  and  evaluation 
conducted  in  as  realistic  an  operational  environment  as  possible  to  determine  the 
operational  effectiveness  and  suitability  of  a  system  under  realistic  combat  conditions 
and  to  determine  if  the  minimum  acceptable  operational  performance  requirements  as 
specified  in  the  ORD  have  been  satisfied.  (DoD  5000.2-RDoDI  5000.2) 

OPERATIONS  SECURITY  (OPSEC)  A  process  of  identifying  critical  information 
and  subsequently  analyzing  friendly  actions  attendant  to  military  operations  and  other 
activities  to:  a.)  Identify  those  actions  that  can  be  observed  by  adversary  intelligence, 

b. )  Determine  indicators  adversary  intelligence  might  obtain  that  could  be  interpreted 
or  pieced  together  to  derive  critical  information  in  time  to  be  useful  to  adversaries,  and 

c. )  Select  and  execute  measures  that  eliminate  or  reduce  to  an  acceptable  level  the 
vulnerabilities  of  friendly  actions  to  adversary  exploitation.  Also  called  OPSEC.  (CJCS 
MOP  30) 

PERFORMANCE  Those  operational  and  support  characteristics  of  the  system  that 
allow  it  to  effectively  and  efficiently  perform  its  assigned  mission  over  time.  The 
support  characteristics  of  the  system  include  both  supportability  aspects  of  the  design 
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and  the  support  elements  necessary  for  system  operatiom  (DoD  5000.2-RBeBI 

5000.2) 

PERFORMANCE  OBJECTIVE  The  performance  parameter  value  beyond  the 
minimum  operational  requirement  that  could  have  a  beneficial  impact  on  achieved 
I  operational  capability^  (DoD  5000.2-RDoDI  5000.2) 

PERFORMANCE  THRESHOLD  The  performance  parameter  value  that  meets  the 
minimum  level  of  system  performance  that  will  satisfy  the  validated  mission  need. 

Also  known  as  the  minimum  acceptable  operational  requirement  (DoD  5000.2-RDeDI 

5000.2) 

PRELIMINARY  DESIGN  REVIEW  (PDR)  A  review  conducted  on  each 
configuration  item  to  evaluate  the  progress,  technical  adequacy,  and  risk  resolution  of 
the  selected  design  approach;  to  determine  its  compatibility  with  performance  and 
engineering  requirements  of  the  development  specification;  and  to  establish  the 
existence  and  compatibility  of  the  physical  and  functional  interfaces  among  the  item  and 
other  items  of  equipment,  facilities,  computer  programs,  and  personnel.  Conducted 
during  Phase  I,  DEM/VAL  (for  prototypes),  and  Phase  11 ,  EMD  (DoD  5000.2-RDeDI 

5000.2) 

PRODUCIBILITY  The  relative  ease  of  manufacturing  an  item  or  system.  The 
relative  ease  is  governed  by  the  characteristics  and  feamres  of  a  design  that  enable 
economical  fabrication,  assembly,  inspection,  and  testing  using  available  manufacturing 
I  techniques^(DoD  5000.2-RDoDI  5000.2) 

PRODUCTION  The  conversion  of  raw  materials  into  products  and/or  components 
thereof,  through  a  series  of  manufacturing  processes.  It  includes  functions  of 
production  engineering,  controlling,  quality  assurance,  and  the  determination  of 
resources  requirements.  (Joint  Pub  1-02) 

PRODUCTION  BASELINE  The  Production  Baseline  will  contain  objectives  and 
thresholds  for  key  cost,  schedule  and  performance  parameters  which  have  been  updated 
from  the  development  baseline.  The  Production  Baseline  will  be  submitted  as  a  stand- 
1  alone  part  of  the  Milestone  III  documentation.  (DoD  5000.2-RDoDI  5000.2) 

PRODUCTION  PROTOTYPE  (PP)  A  final  model  of  a  design  before  the  pilot  unit  is 
approved  for  production.  It  should  be  highly  representative  of  final  equipment,  except 
that  the  exact  manufacturing  assembly  process  and  production  design  changes  may  not 
yet  be  used  or  incorporated.  It  is  suitable  for  complete  evaluation  of  its  electrical 
and/or  mechanical  form  and  may  be  in  a  brassboard  (technical  and  operational 
configuration.  (EW  T&E  Task  Force) 


PROGRAM  EXECUTIVE  OFFICER  (PEO)  A  military  or  civilian  official  who  has 
primary  responsibility  for  directing  several  acquisition  category  I  programs  and  for 
assigned  ACAT  U,  IH,  and  IV  programs.  tPoD  5000.2-RDoDI  5000.2) 

PROGRAM  MANAGER  (PM)  A  military  or  civilian  official  who  is  responsible  for 
managing  an  acquisition  program.  (BoD  5000.2-RDoDI  5000.2) 

PROTOTYPE  A  model  suitable  for  evaluation  of  design,  performance,  and 
production  potential.  (Joint  Pub  1-02) 

PSYCHOLOGICAL  OPERATIONS  (PSYOP)  Planned  operations  to  convey  selected 
mformation  and  indicators  to  foreign  audiences  to  influence  their  emotions,  motive, 
objective  reasonmg,  and  ultunately  the  behavior  of  foreign  governments,  organizations, 
groups,  and  individuals.  The  purpose  of  PSYOP  is  to  induce  or  reinforce  foreign 
attitude  and  behavior  favorable  to  the  originator's  objectives.  (CJCS  MOP  30) 

RELIABILITY  The  ability  of  a  system  and  its  parts  to  perform  its  mission  without 
failure,  degradation,  or  demand  on  the  support  system  (DoDI  5000-2-R) 

RELIANCE  STUDY  The  "Reliance"  process  for  T&E  is  an  outgrowth  of  the  DoD 
Management  Report  Decision  of  November  1990  whose  objectives  were  to  "right-size" 
the  T&E  infrastructure  for  the  fiimre,  to  eliminate  unwarranted  duplication  and  to 
count,  in  the  future,  on  inter-service  "reliance"  for  non-service-peculiar  T&E  support 
capabilities.  EW  T&E  is  a  key  element  of  that  study  effort.  (Reliance  Study) 

REQUIREMENT  An  established  need  justifying  the  timely  allocation  of  resources  to 
achieve  a  capability  to  accomplish  approved  military  objectives,  missions,  or  tasks. 
(Joint  Pub  1-02) 

REQUIREMENTS  CORRELATION  MATRIX  (RCM)  A  management  tool  used  to 
provide  a  system  audit  trail.  It  contains  a  comparison  of  the  user's  needs,  system 
requirements,  contractual  specifications,  and  operational  evaluation  criteria.  (EW  T&E 
Task  Force) 

RISK  A  subjective  assessment  made  regarding  the  likelihood  or  probability  of  not 
achieving  a  specific  objective  by  the  time  established  with  the  resources  provided  or 
requested.  It  also  refers  to  overall  program  risk.  (DoD  5000.2-RDoDI  5000.2) 

SELF-PROTECTION  EW  systems  which  are  integrated  or  carried  on  board  a  host 
platform  and  provide  platform  self-protection  through  active  transmission  or  reflection 
of  electromagnetic  energy  or  destruction  of  enemy  command,  control,  and 
communications  systems.  (DoD  EW  PLAN) 


SIMULATION  Simulation  is  a  method  for  implementing  a  model.  It  is  the  process  of 
conducting  experiments  with  a  model  for  the  purpose  of  understanding  the  behavior  of 
the  system  modeled  under  selected  conditions  or  of  evaluating  various  strategies  for  the 
operation  of  the  system  within  the  limits  imposed  by  developmental  or  operational 
criteria.  (BoD  5000.2-RDoDI  5000:2) 

SPECIFICATIONS  Contractual  values  that  reflect  the  expected  capabilities  to  be 
produced  and/or  fielded  and  that  are  traceable  to  the  cost,  schedule,  and  performance 
objectives  of  the  acquisition  program  baseline.  They  are  also  tied  to  the  acquisition 
phase  in  which  the  program  is  currently  engaged  and  reflect  the  demonstration 
requirements  is  support  of  exit  criteria.  (BoD  5000.2-RoDI  5000.2) 

SPECTRUM  MANAGEMENT  Planning,  coordinating,  and  managing  joint  use  of  the 
electromagnetic  spectrum  through  operational,  engineering,  and  administrative 
procedures,  with  the  objective  of  enabling  electronic  systems  to  perform  their  functions 
in  the  intended  environment  without  causing  or  suffering  unacceptable  interference. 
(CJCS  MOP  6) 

SUPPORTABILITY  The  degree  to  which  system  design  characteristics  and  planned 
logistics  resources,  meet  system  peacetime  readiness  and  wartime  utilization 
requirements.  CBoB  5000.2-RBoBI  5000.2) 

SUPPRESSION  OF  ENEMY  AIR  DEFENSES  (SEAD)  That  activity  which 
neutralizes,  destroys,  or  temporarily  degrades  enemy  air  defenses  in  a  specific  area  by 
physical  attack,  deception,  and/or  electronic  warfare.  (CJCS  MOP  6) 

SURGE  An  increase  in  the  production  or  repair  of  defense  goods  of  limited  duration. 
(BoB  5000.2-RBoBI  5000.-2) 

SURVIVABILITY  The  capability  of  a  system  to  avoid  or  withstand  man-made  hostile 
environments  without  suffering  an  abortive  impairment  of  its  ability  to  accomplish  its 
designated  mission^tBoB  5000.2-RBoBI  5000.2) 

SUSCEPTIBILITY  The  degree  to  which  a  device,  equipment,  or  weapon  system  is 
open  to  effective  attack  due  to  one  or  more  inherent  weaknesses.  (Susceptibility  is  a 
function  of  operational  tactics,  countermeasures,  probability  of  enemy  fielding  a  threat, 
etc.U  Susceptibility  is  considered  a  subset  of  survivability.  (BoB  5000.2-RBoBI  500&- 

S) 

SYNTHESIS  In  the  T&E  Process  usage,  synthesis  is  the  combining  and  examining  of 
processed  information  (Step  Three)  with  expected  outcomes  (Step  Two)  and  other 
information  using  both  technical  and  operational  judgment.  (EW  T&E  Task  Force) 
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SYSTEM  Any  organized  assembly  of  resources  and  procedures  united  and  regulated 
by  interaction  or  interdependence  to  accomplish  a  set  of  specific  function.  (Joint  Pub  1- 
02) 

SYSTEM  EFFECTIVENESS  A  measure  of  the  extent  to  which  a  system  may  be 
expected  to  achieve  a  set  of  specific  mission  requirements  expressed  as  a  function  of 
availability,  dependability,  and  capability.  (EW  T&E  Task  Force) 

SYSTEM  MATURITY  MATRIX  An  acquisition  management  tool  that  can  be  used  to 
highlight  differences  between  the  required  objective  and/or  threshold  values  and  the 
demonstrated  values  resulting  from  scheduled  testing.  (EW  T&E  Task  Force) 

SYSTEM  PERFORMANCE  PARAMETER  (SPP)  A  measurable  property  of  a 
system  concept,  a  system  design,  or  a  system  configuration  that  characterizes  system 
performance,  e.g.,  speed  in  knots  or  transmitted  power  in  watts,  that  can  be  directly 
measured  by  instrumenting  the  system  under  test.  (EW  T&E  Task  Force) 

SYSTEM  RELIABILITY  AND  MAINTAINABILITY  PARAMETER  A  measure 
of  reliability  or  maintainability  in  which  the  units  of  measurement  are  directly  related  to 
operational  readiness,  mission  success,  maintenance  manpower  or  logistic  support  cost. 
rPoD  5000.2-RDoDI  5000.2) 

SYSTEM  THREAT  ASSESSMENT  A  systems-specific  assessment  that  describes  the 
threat  to  be  countered  and  the  projected  threat  environment.  The  threat  assessment  will 
be  derived  from  DIA  produced  or  validated  documents.  (DoD  5000.2-RDoDI  5000.2) 

SYSTEMS  ENGINEERING  A  comprehensive,  iterative  technical  management 
process  to:  1)  Translate  an  operational  need  into  a  configured  system  meeting  that  need 
through  a  systematic,  concurrent  approach  to  integrated  design  of  the  system  and  its 
related  manufacturing,  test,  and  support  processes;  2)  Integrate  the  technical  inputs  of 
the  entire  development  community  and  all  technical  disciplines  into  a  coordinated  effort 
that  meets  established  program  cost,  schedule,  and  performance  objectives;  3)  Ensure 
the  compatibility  of  all  functional  and  physical  interfaces  and  ensure  that  system 
definition  and  design  reflect  the  requirements  for  all  system  elements;  4)  Characterize 
technical  risks,  develop  risk  abatement  approaches,  and  reduce  technical  risk  through 
early  test  and  demonstration  of  system  elements.  (DoD  5000.2-RDoDI  5000t3) 

TECHNICAL  DATA  Scientific  or  technical  information  recorded  in  any  form  or 
medium  (such  as  manuals  and  drawings).  Computer  programs  and  related  software  are 
not  technical  data;  documentation  of  computer  programs  and  related  software  are.  Also 
excluded  are  financial  data  or  other  information  related  to  contract  administration. 

(DoD  5000.2-RDoDI  5000.-2) 


TECHNICAL  PERFORMANCE  PARAMETER  (TPP)  A  selected  subset  of  system- 
specific  performance  parameters  used  as  the  technical  measures  tracked  in  the  systems 
engineering  technical  performance  measurement  program.  TPPs  are  used  to  measure 
compliance  with  requirements  and  to  assess  the  level  of  technical  risk  in  a  development 
program.  (EW  T&E  Task  Force) 

TECHNICAL  REQUIREMENTS  FORMULATION  The  proeess  of  converting 
operational  requirements  into  technical  requirements  that  can  be  acted  on  by  designers. 
(DSMC) 

TEST  AND  EVALUATION  (T&E)  Any  project  or  program  designed  to  obtain, 
verify,  and  provide  data  for  the  evaluation  of  research  and  development  other  than 
laboratory  experiments  for  the  purpose  of  determining  if  the  minimum  acceptable 
operational  performance  requirements  as  specified  in  the  Operational  Requirements 
I  Document  have  been  satisfied.  (DoD  5000.2-RDoDI  5000.2) 

TEST  AND  EVALUATION  MASTER  PLAN  (TEMP)  An  overall  T&E  plan 
designed  to  identify  and  integrate  the  efforts  and  schedules  of  all  T&E  to  be  done  in  an 
I  acquisition  program.  tPoD  5000.2-RDoDI  5000.2) 

TESTBED  A  system  representation  consisting  partially  of  acmal  hardware  and/or 
software  and  partially  of  computer  models  or  prototype  hardware  and/or  software. 

I  (DoD  5000.2-RDoDI  5000.2). 

TEST  CONDITIONS  The  environment  (e.g.,  location,  weather),  scenario,  and 
operating  procedures  and  configurations  for  the  SUT  and  adversaries  in  the  test 
scenario.  (EW  T&E  Task  Force) 

TEST  ENVIRONMENT  The  test  location,  facility  type,  weather  conditions,  threat, 
electromagnetic  and  stimulation  environments,  etc.,  under  whieh  the  test  is  conducted. 
(EW  T&E  Task  Force) 

TEST  EVENT  An  activity  during  conduct  of  a  test  trial  that  requires  a  response  by  the 
system  and/or  personnel  under  test.  (EW  T&E  Task  Force) 

TEST  OBJECTIVE  The  specific  performance  or  technical  parameters  to  be  measured 
during  the  test  to  evaluate  system  performance,  system  operational  effectiveness,  or 
system  suitability.  (EW  T&E  Task  Force) 

TEST  RESOURCES  A  collective  term  that  encompasses  all  elements  necessary  to 
plan,  conduct,  and  collect/analyze  data  from  a  test  event  or  program.  (AR  73-1) 

TEST  SCENARIO  A  simation,  representative  of  what  the  system  under  test  may 
encounter  in  real  life,  that  is  used  to  enact  a  set  of  events  between  it  and  adversaries 


included  in  the  situation  (e.g.,  threat  simulator  locations  and  flight  profiles).  (EW  T&E 
Task  Force) 

TESTBED  A  system  representation  consisting  partially  of  actual  hardware  and/or 
software  and  partially  of  computer  models  or  prototype  hardware  and/or  software. 
tPoD  5000.2-RDoDI  5000:^) 

THREAT  Current  and  future  capabilities  of  a  potential  enemy  force  against  one  or 
more  US  developmental  systems  in  terms  of  combat  materiel,  employment,  doctrine, 
force  structure,  and  combat  environment.  (AR  73-1) 

THRESHOLD  A  minimum  acceptable  value  for  a  performance  parameter  which,  in 
the  user's  judgment,  is  necessary  to  provide  an  operational  capability  that  will  satisfy 
the  mission  need.  The  threshold  must  be  met  in  order  to  gain  approval  from  the 
milestone  decision  authority.  (DoD  50Q0.2-RDoDI  5()00.-2) 

UPGRADE  A  change  to  a  system  (whether  for  safety,  to  correct  a  deficiency,  or  to 
improve  program  performance)  to  a  system  that  is  out  of  production.  Upgrades  are  part 
of  the  Milestone  0  decision  process.  (DoD  5000.2-RDoDI  5000:3) 

VALIDATION  The  process  of  determining  the  degree  to  which  a  model  is  an  accurate 
representation  of  the  real-world  from  the  perspective  of  the  intended  uses  of  the  model. 
^oDD  5000.59) 

VALUE  An  assigned  or  calculated  numerical  quantity.  An  amount  considered  to  be  a 
suitable  equivalent  for  something  else.  (AMERICAN  HERITAGE  DICTIONARY) 

VERIFICATION  The  process  of  determining  that  a  model  implementation  accurately 
represents  the  developers'  conceptual  description  and  specifications.  (DoDD  5000.59) 

VETRONICS  Electrical  and  electronic  systems  aboard  ground  vehicles.  (Commonly 
used  military  jargon  that  is  a  contraction  of  vehicle  electronics)  (EW  T&E  Task  Force) 

VULNERABILITY  The  characteristics  of  a  system  that  cause  it  to  suffer  a  definite 
degradation  (loss  or  reduction  of  capability  to  perform  the  designated  mission)  as  a 
result  of  having  been  subjected  to  a  certain  (defined)  level  of  effects  in  an  unnatural 
(man-made)  hostile  environment.  Vulnerability  is  considered  a  subset  of  survivability. 
fPoD  5000.2-RDoDI  5000.2) 

WARTIME  RESERVE  MODES  (WARM)  Characteristics  and  operating  procedures 
of  sensor,  communications,  navigation  aids,  threat  recognition,  weapons,  and 
countermeasures  systems  that  (a)  will  contribute  to  military  effectiveness  if  unknown  to 
or  misunderstood  by  opposing  commanders  before  they  are  used,  but  (b)  could  be 
exploited  or  neutralized  if  known  in  advance.  Wartime  reserve  modes  are  deliberately 
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held  in  reserve  for  wartime  or  emergency  use  and  seldom,  if  ever,  applied  or 
intercepted  prior  to  such  use.  (CJCS  MOP  6) 
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